2026-02-25T22:15:07.335 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-25T22:15:07.349 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T22:15:07.424 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/emzhang-2026-02-25_22:10:53-rados:standalone-wip-mon-colocate-distro-default-trial/70699 branch: wip-mon-colocate description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} email: null first_in_suite: false flavor: default job_id: '70699' kernel: branch: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: trial name: emzhang-2026-02-25_22:10:53-rados:standalone-wip-mon-colocate-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-mon-colocate ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd workunit: branch: wip-mon-colocate-test sha1: 61d4431d3ec879bd98659388b3655acf3d392418 owner: scheduled_emzhang@soko04.front.sepia.ceph.com priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1952 sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd sleep_before_teardown: 0 suite: rados:standalone suite_branch: wip-mon-colocate-test suite_path: /home/teuthworker/src/github.com_Ericmzhang_ceph_61d4431d3ec879bd98659388b3655acf3d392418/qa suite_relpath: qa suite_repo: https://github.com/Ericmzhang/ceph.git suite_sha1: 61d4431d3ec879bd98659388b3655acf3d392418 targets: trial125.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIQvH6pwJMIM7BynouG9U+RYVxjCxA0VHzMfJs0+7LtZB5ZvN4R+/Jogt/vXA6zOceDU5VCcKaPLCKsGxEXAtx8= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-25_22:10:53 tube: trial user: emzhang verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T22:15:07.425 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_Ericmzhang_ceph_61d4431d3ec879bd98659388b3655acf3d392418/qa; will attempt to use it 2026-02-25T22:15:07.425 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_Ericmzhang_ceph_61d4431d3ec879bd98659388b3655acf3d392418/qa/tasks 2026-02-25T22:15:07.425 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T22:15:07.426 INFO:teuthology.task.internal:Checking packages... 2026-02-25T22:15:07.427 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd' 2026-02-25T22:15:07.427 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T22:15:07.427 INFO:teuthology.packaging:ref: None 2026-02-25T22:15:07.427 INFO:teuthology.packaging:tag: None 2026-02-25T22:15:07.427 INFO:teuthology.packaging:branch: wip-mon-colocate 2026-02-25T22:15:07.427 INFO:teuthology.packaging:sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T22:15:07.427 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-mon-colocate 2026-02-25T22:15:07.545 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5305-g0886691e-1jammy 2026-02-25T22:15:07.546 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T22:15:07.557 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T22:15:07.557 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T22:15:07.571 INFO:teuthology.task.internal:Saving configuration 2026-02-25T22:15:07.575 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T22:15:07.585 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T22:15:07.641 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial125.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/emzhang-2026-02-25_22:10:53-rados:standalone-wip-mon-colocate-distro-default-trial/70699', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 22:11:44.645723', 'locked_by': 'scheduled_emzhang@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP2o8UuIcMwMpnM2GYv7Ia4Y3oPzMqmE5ZxXA57o4pmC'} 2026-02-25T22:15:07.641 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T22:15:07.652 INFO:teuthology.task.internal:roles: ubuntu@trial125.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-25T22:15:07.652 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T22:15:07.722 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7c74a1715c60>, signals=[15]) 2026-02-25T22:15:07.722 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T22:15:07.733 INFO:teuthology.task.internal:Opening connections... 2026-02-25T22:15:07.733 DEBUG:teuthology.task.internal:connecting to ubuntu@trial125.front.sepia.ceph.com 2026-02-25T22:15:07.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T22:15:07.794 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T22:15:07.808 DEBUG:teuthology.orchestra.run.trial125:> uname -m 2026-02-25T22:15:07.871 INFO:teuthology.orchestra.run.trial125.stdout:x86_64 2026-02-25T22:15:07.871 DEBUG:teuthology.orchestra.run.trial125:> cat /etc/os-release 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:NAME="Ubuntu" 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:VERSION_ID="22.04" 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:VERSION_CODENAME=jammy 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:ID=ubuntu 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:ID_LIKE=debian 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-25T22:15:07.920 INFO:teuthology.orchestra.run.trial125.stdout:UBUNTU_CODENAME=jammy 2026-02-25T22:15:07.921 INFO:teuthology.lock.ops:Updating trial125.front.sepia.ceph.com on lock server 2026-02-25T22:15:07.992 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T22:15:08.004 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T22:15:08.020 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T22:15:08.020 DEBUG:teuthology.orchestra.run.trial125:> test '!' -e /home/ubuntu/cephtest 2026-02-25T22:15:08.023 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T22:15:08.033 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T22:15:08.033 DEBUG:teuthology.orchestra.run.trial125:> test -z $(ls -A /var/lib/ceph) 2026-02-25T22:15:08.069 INFO:teuthology.orchestra.run.trial125.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-25T22:15:08.069 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T22:15:08.137 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T22:15:08.155 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T22:15:08.155 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T22:15:08.155 DEBUG:teuthology.orchestra.run.trial125:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T22:15:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T22:15:08.158 DEBUG:teuthology.orchestra.run.trial125:> uname -r 2026-02-25T22:15:08.208 INFO:teuthology.orchestra.run.trial125.stdout:5.15.0-170-generic 2026-02-25T22:15:08.209 INFO:teuthology.task.kernel:Running kernel on trial125: 5.15.0-170-generic 2026-02-25T22:15:08.209 DEBUG:teuthology.orchestra.run.trial125:> sudo apt-get clean 2026-02-25T22:15:08.267 DEBUG:teuthology.orchestra.run.trial125:> sudo apt-get update 2026-02-25T22:15:08.534 INFO:teuthology.orchestra.run.trial125.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T22:15:08.543 INFO:teuthology.orchestra.run.trial125.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-25T22:15:08.585 INFO:teuthology.orchestra.run.trial125.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-25T22:15:08.605 INFO:teuthology.orchestra.run.trial125.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-25T22:15:08.678 INFO:teuthology.orchestra.run.trial125.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3274 kB] 2026-02-25T22:15:08.747 INFO:teuthology.orchestra.run.trial125.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [498 kB] 2026-02-25T22:15:08.750 INFO:teuthology.orchestra.run.trial125.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.2 kB] 2026-02-25T22:15:08.751 INFO:teuthology.orchestra.run.trial125.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-25T22:15:08.758 INFO:teuthology.orchestra.run.trial125.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-25T22:15:08.760 INFO:teuthology.orchestra.run.trial125.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-25T22:15:08.761 INFO:teuthology.orchestra.run.trial125.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5400 kB] 2026-02-25T22:15:08.788 INFO:teuthology.orchestra.run.trial125.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1025 kB] 2026-02-25T22:15:08.791 INFO:teuthology.orchestra.run.trial125.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [3007 kB] 2026-02-25T22:15:08.803 INFO:teuthology.orchestra.run.trial125.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [430 kB] 2026-02-25T22:15:08.804 INFO:teuthology.orchestra.run.trial125.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-25T22:15:08.804 INFO:teuthology.orchestra.run.trial125.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1019 kB] 2026-02-25T22:15:08.808 INFO:teuthology.orchestra.run.trial125.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [225 kB] 2026-02-25T22:15:08.808 INFO:teuthology.orchestra.run.trial125.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.6 kB] 2026-02-25T22:15:08.809 INFO:teuthology.orchestra.run.trial125.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5217 kB] 2026-02-25T22:15:08.827 INFO:teuthology.orchestra.run.trial125.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [993 kB] 2026-02-25T22:15:12.690 INFO:teuthology.orchestra.run.trial125.stdout:Fetched 23.1 MB in 2s (13.0 MB/s) 2026-02-25T22:15:13.209 INFO:teuthology.orchestra.run.trial125.stdout:Reading package lists... 2026-02-25T22:15:13.222 DEBUG:teuthology.orchestra.run.trial125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-25T22:15:13.254 INFO:teuthology.orchestra.run.trial125.stdout:Reading package lists... 2026-02-25T22:15:13.361 INFO:teuthology.orchestra.run.trial125.stdout:Building dependency tree... 2026-02-25T22:15:13.362 INFO:teuthology.orchestra.run.trial125.stdout:Reading state information... 2026-02-25T22:15:13.448 INFO:teuthology.orchestra.run.trial125.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T22:15:13.448 INFO:teuthology.orchestra.run.trial125.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T22:15:13.448 INFO:teuthology.orchestra.run.trial125.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-25T22:15:13.448 INFO:teuthology.orchestra.run.trial125.stdout: sg3-utils-udev 2026-02-25T22:15:13.448 INFO:teuthology.orchestra.run.trial125.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T22:15:13.448 INFO:teuthology.orchestra.run.trial125.stdout:The following additional packages will be installed: 2026-02-25T22:15:13.448 INFO:teuthology.orchestra.run.trial125.stdout: linux-generic linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-25T22:15:13.448 INFO:teuthology.orchestra.run.trial125.stdout: linux-headers-generic linux-image-5.15.0-171-generic 2026-02-25T22:15:13.448 INFO:teuthology.orchestra.run.trial125.stdout: linux-modules-5.15.0-171-generic linux-modules-extra-5.15.0-171-generic 2026-02-25T22:15:13.449 INFO:teuthology.orchestra.run.trial125.stdout:Suggested packages: 2026-02-25T22:15:13.449 INFO:teuthology.orchestra.run.trial125.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2026-02-25T22:15:13.461 INFO:teuthology.orchestra.run.trial125.stdout:The following NEW packages will be installed: 2026-02-25T22:15:13.461 INFO:teuthology.orchestra.run.trial125.stdout: linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-25T22:15:13.461 INFO:teuthology.orchestra.run.trial125.stdout: linux-image-5.15.0-171-generic linux-modules-5.15.0-171-generic 2026-02-25T22:15:13.461 INFO:teuthology.orchestra.run.trial125.stdout: linux-modules-extra-5.15.0-171-generic 2026-02-25T22:15:13.461 INFO:teuthology.orchestra.run.trial125.stdout:The following packages will be upgraded: 2026-02-25T22:15:13.461 INFO:teuthology.orchestra.run.trial125.stdout: linux-generic linux-headers-generic linux-image-generic 2026-02-25T22:15:13.869 INFO:teuthology.orchestra.run.trial125.stdout:3 upgraded, 5 newly installed, 0 to remove and 31 not upgraded. 2026-02-25T22:15:13.869 INFO:teuthology.orchestra.run.trial125.stdout:Need to get 113 MB of archives. 2026-02-25T22:15:13.869 INFO:teuthology.orchestra.run.trial125.stdout:After this operation, 585 MB of additional disk space will be used. 2026-02-25T22:15:13.869 INFO:teuthology.orchestra.run.trial125.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-171-generic amd64 5.15.0-171.181 [22.7 MB] 2026-02-25T22:15:15.218 INFO:teuthology.orchestra.run.trial125.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-171-generic amd64 5.15.0-171.181 [11.6 MB] 2026-02-25T22:15:15.546 INFO:teuthology.orchestra.run.trial125.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-171-generic amd64 5.15.0-171.181 [63.9 MB] 2026-02-25T22:15:17.766 INFO:teuthology.orchestra.run.trial125.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.171.160 [1658 B] 2026-02-25T22:15:17.854 INFO:teuthology.orchestra.run.trial125.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.171.160 [2486 B] 2026-02-25T22:15:17.953 INFO:teuthology.orchestra.run.trial125.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171 all 5.15.0-171.181 [12.4 MB] 2026-02-25T22:15:18.945 INFO:teuthology.orchestra.run.trial125.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171-generic amd64 5.15.0-171.181 [2849 kB] 2026-02-25T22:15:19.061 INFO:teuthology.orchestra.run.trial125.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.171.160 [2332 B] 2026-02-25T22:15:19.259 INFO:teuthology.orchestra.run.trial125.stdout:Fetched 113 MB in 6s (20.3 MB/s) 2026-02-25T22:15:19.280 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package linux-modules-5.15.0-171-generic. 2026-02-25T22:15:19.452 INFO:teuthology.orchestra.run.trial125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-25T22:15:19.453 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../0-linux-modules-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T22:15:19.455 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:19.800 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package linux-image-5.15.0-171-generic. 2026-02-25T22:15:19.808 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../1-linux-image-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T22:15:19.818 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:19.862 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-171-generic. 2026-02-25T22:15:19.870 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T22:15:19.871 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:20.960 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../3-linux-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T22:15:20.961 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking linux-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T22:15:20.986 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T22:15:20.987 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking linux-image-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T22:15:21.003 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package linux-headers-5.15.0-171. 2026-02-25T22:15:21.012 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../5-linux-headers-5.15.0-171_5.15.0-171.181_all.deb ... 2026-02-25T22:15:21.013 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-25T22:15:22.375 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package linux-headers-5.15.0-171-generic. 2026-02-25T22:15:22.389 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../6-linux-headers-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T22:15:22.390 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:22.981 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T22:15:22.982 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking linux-headers-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T22:15:23.008 INFO:teuthology.orchestra.run.trial125.stdout:Setting up linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-25T22:15:23.009 INFO:teuthology.orchestra.run.trial125.stdout:Setting up linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:23.010 INFO:teuthology.orchestra.run.trial125.stdout:Setting up linux-headers-generic (5.15.0.171.160) ... 2026-02-25T22:15:23.012 INFO:teuthology.orchestra.run.trial125.stdout:Setting up linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:23.795 INFO:teuthology.orchestra.run.trial125.stdout:I: /boot/vmlinuz.old is now a symlink to vmlinuz-5.15.0-170-generic 2026-02-25T22:15:23.795 INFO:teuthology.orchestra.run.trial125.stdout:I: /boot/initrd.img.old is now a symlink to initrd.img-5.15.0-170-generic 2026-02-25T22:15:23.795 INFO:teuthology.orchestra.run.trial125.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-171-generic 2026-02-25T22:15:23.795 INFO:teuthology.orchestra.run.trial125.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-171-generic 2026-02-25T22:15:23.803 INFO:teuthology.orchestra.run.trial125.stdout:Setting up linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:24.610 INFO:teuthology.orchestra.run.trial125.stdout:Setting up linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:25.410 INFO:teuthology.orchestra.run.trial125.stdout:Setting up linux-image-generic (5.15.0.171.160) ... 2026-02-25T22:15:25.411 INFO:teuthology.orchestra.run.trial125.stdout:Setting up linux-generic (5.15.0.171.160) ... 2026-02-25T22:15:25.412 INFO:teuthology.orchestra.run.trial125.stdout:Processing triggers for linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:25.417 INFO:teuthology.orchestra.run.trial125.stdout:/etc/kernel/postinst.d/initramfs-tools: 2026-02-25T22:15:25.417 INFO:teuthology.orchestra.run.trial125.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-171-generic 2026-02-25T22:15:32.472 INFO:teuthology.orchestra.run.trial125.stdout:W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast 2026-02-25T22:15:39.036 INFO:teuthology.orchestra.run.trial125.stdout:/etc/kernel/postinst.d/zz-update-grub: 2026-02-25T22:15:39.037 INFO:teuthology.orchestra.run.trial125.stdout:Sourcing file `/etc/default/grub' 2026-02-25T22:15:39.056 INFO:teuthology.orchestra.run.trial125.stdout:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-25T22:15:39.057 INFO:teuthology.orchestra.run.trial125.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-25T22:15:39.059 INFO:teuthology.orchestra.run.trial125.stdout:Generating grub configuration file ... 2026-02-25T22:15:39.142 INFO:teuthology.orchestra.run.trial125.stdout:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-25T22:15:39.149 INFO:teuthology.orchestra.run.trial125.stdout:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-25T22:15:39.325 INFO:teuthology.orchestra.run.trial125.stdout:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-25T22:15:39.330 INFO:teuthology.orchestra.run.trial125.stdout:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-25T22:15:39.372 INFO:teuthology.orchestra.run.trial125.stdout:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-25T22:15:39.378 INFO:teuthology.orchestra.run.trial125.stdout:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-25T22:15:39.452 INFO:teuthology.orchestra.run.trial125.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-25T22:15:39.452 INFO:teuthology.orchestra.run.trial125.stdout:Systems on them will not be added to the GRUB boot configuration. 2026-02-25T22:15:39.452 INFO:teuthology.orchestra.run.trial125.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-25T22:15:39.455 INFO:teuthology.orchestra.run.trial125.stdout:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T22:15:39.463 INFO:teuthology.orchestra.run.trial125.stdout:done 2026-02-25T22:15:39.874 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:15:39.874 INFO:teuthology.orchestra.run.trial125.stdout:Pending kernel upgrade! 2026-02-25T22:15:39.874 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:15:39.874 INFO:teuthology.orchestra.run.trial125.stdout:Running kernel version: 2026-02-25T22:15:39.874 INFO:teuthology.orchestra.run.trial125.stdout: 5.15.0-170-generic 2026-02-25T22:15:39.874 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:15:39.874 INFO:teuthology.orchestra.run.trial125.stdout:Diagnostics: 2026-02-25T22:15:39.875 INFO:teuthology.orchestra.run.trial125.stdout: The currently running kernel version is not the expected kernel version 5.15.0-171-generic. 2026-02-25T22:15:39.875 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:15:39.875 INFO:teuthology.orchestra.run.trial125.stdout:Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting. [Return] 2026-02-25T22:15:39.875 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:15:39.875 INFO:teuthology.orchestra.run.trial125.stdout:The processor microcode seems to be up-to-date. 2026-02-25T22:15:39.875 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:15:39.875 INFO:teuthology.orchestra.run.trial125.stdout:No services need to be restarted. 2026-02-25T22:15:39.875 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:15:39.875 INFO:teuthology.orchestra.run.trial125.stdout:No containers need to be restarted. 2026-02-25T22:15:39.875 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:15:39.875 INFO:teuthology.orchestra.run.trial125.stdout:No user sessions are running outdated binaries. 2026-02-25T22:15:39.876 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:15:39.876 INFO:teuthology.orchestra.run.trial125.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-25T22:15:40.714 DEBUG:teuthology.orchestra.run.trial125:> dpkg -s linux-image-generic 2026-02-25T22:15:40.724 INFO:teuthology.orchestra.run.trial125.stdout:Package: linux-image-generic 2026-02-25T22:15:40.724 INFO:teuthology.orchestra.run.trial125.stdout:Status: install ok installed 2026-02-25T22:15:40.724 INFO:teuthology.orchestra.run.trial125.stdout:Priority: optional 2026-02-25T22:15:40.725 INFO:teuthology.orchestra.run.trial125.stdout:Section: kernel 2026-02-25T22:15:40.725 INFO:teuthology.orchestra.run.trial125.stdout:Installed-Size: 22 2026-02-25T22:15:40.725 INFO:teuthology.orchestra.run.trial125.stdout:Maintainer: Ubuntu Kernel Team 2026-02-25T22:15:40.725 INFO:teuthology.orchestra.run.trial125.stdout:Architecture: amd64 2026-02-25T22:15:40.725 INFO:teuthology.orchestra.run.trial125.stdout:Source: linux-meta 2026-02-25T22:15:40.725 INFO:teuthology.orchestra.run.trial125.stdout:Version: 5.15.0.171.160 2026-02-25T22:15:40.725 INFO:teuthology.orchestra.run.trial125.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-25T22:15:40.725 INFO:teuthology.orchestra.run.trial125.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-25T22:15:40.725 INFO:teuthology.orchestra.run.trial125.stdout:Recommends: thermald 2026-02-25T22:15:40.725 INFO:teuthology.orchestra.run.trial125.stdout:Description: Generic Linux kernel image 2026-02-25T22:15:40.725 INFO:teuthology.orchestra.run.trial125.stdout: This package will always depend on the latest generic kernel image 2026-02-25T22:15:40.726 INFO:teuthology.orchestra.run.trial125.stdout: available. 2026-02-25T22:15:40.726 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-170-generic Expected: 5.15.0-171-generic 2026-02-25T22:15:40.726 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T22:15:40.726 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T22:15:40.726 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T22:15:40.726 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial125.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T22:15:40.727 DEBUG:teuthology.orchestra.run.trial125:> sudo apt-get clean 2026-02-25T22:15:40.809 DEBUG:teuthology.orchestra.run.trial125:> sudo apt-get update 2026-02-25T22:15:40.991 INFO:teuthology.orchestra.run.trial125.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T22:15:40.996 INFO:teuthology.orchestra.run.trial125.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-25T22:15:41.006 INFO:teuthology.orchestra.run.trial125.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-25T22:15:41.017 INFO:teuthology.orchestra.run.trial125.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-25T22:15:41.837 INFO:teuthology.orchestra.run.trial125.stdout:Reading package lists... 2026-02-25T22:15:41.852 DEBUG:teuthology.orchestra.run.trial125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-25T22:15:41.885 INFO:teuthology.orchestra.run.trial125.stdout:Reading package lists... 2026-02-25T22:15:41.993 INFO:teuthology.orchestra.run.trial125.stdout:Building dependency tree... 2026-02-25T22:15:41.994 INFO:teuthology.orchestra.run.trial125.stdout:Reading state information... 2026-02-25T22:15:42.079 INFO:teuthology.orchestra.run.trial125.stdout:linux-image-generic is already the newest version (5.15.0.171.160). 2026-02-25T22:15:42.079 INFO:teuthology.orchestra.run.trial125.stdout:linux-image-generic set to manually installed. 2026-02-25T22:15:42.079 INFO:teuthology.orchestra.run.trial125.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T22:15:42.079 INFO:teuthology.orchestra.run.trial125.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T22:15:42.079 INFO:teuthology.orchestra.run.trial125.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-25T22:15:42.079 INFO:teuthology.orchestra.run.trial125.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-25T22:15:42.080 INFO:teuthology.orchestra.run.trial125.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-25T22:15:42.080 INFO:teuthology.orchestra.run.trial125.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-25T22:15:42.080 INFO:teuthology.orchestra.run.trial125.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T22:15:42.099 INFO:teuthology.orchestra.run.trial125.stdout:0 upgraded, 0 newly installed, 0 to remove and 31 not upgraded. 2026-02-25T22:15:42.101 DEBUG:teuthology.orchestra.run.trial125:> dpkg -s linux-image-generic 2026-02-25T22:15:42.150 INFO:teuthology.orchestra.run.trial125.stdout:Package: linux-image-generic 2026-02-25T22:15:42.150 INFO:teuthology.orchestra.run.trial125.stdout:Status: install ok installed 2026-02-25T22:15:42.150 INFO:teuthology.orchestra.run.trial125.stdout:Priority: optional 2026-02-25T22:15:42.150 INFO:teuthology.orchestra.run.trial125.stdout:Section: kernel 2026-02-25T22:15:42.150 INFO:teuthology.orchestra.run.trial125.stdout:Installed-Size: 22 2026-02-25T22:15:42.150 INFO:teuthology.orchestra.run.trial125.stdout:Maintainer: Ubuntu Kernel Team 2026-02-25T22:15:42.150 INFO:teuthology.orchestra.run.trial125.stdout:Architecture: amd64 2026-02-25T22:15:42.150 INFO:teuthology.orchestra.run.trial125.stdout:Source: linux-meta 2026-02-25T22:15:42.151 INFO:teuthology.orchestra.run.trial125.stdout:Version: 5.15.0.171.160 2026-02-25T22:15:42.151 INFO:teuthology.orchestra.run.trial125.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-25T22:15:42.151 INFO:teuthology.orchestra.run.trial125.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-25T22:15:42.151 INFO:teuthology.orchestra.run.trial125.stdout:Recommends: thermald 2026-02-25T22:15:42.151 INFO:teuthology.orchestra.run.trial125.stdout:Description: Generic Linux kernel image 2026-02-25T22:15:42.151 INFO:teuthology.orchestra.run.trial125.stdout: This package will always depend on the latest generic kernel image 2026-02-25T22:15:42.151 INFO:teuthology.orchestra.run.trial125.stdout: available. 2026-02-25T22:15:42.152 DEBUG:teuthology.orchestra.run.trial125:> mktemp 2026-02-25T22:15:42.197 INFO:teuthology.orchestra.run.trial125.stdout:/tmp/tmp.kMjPrtCKwF 2026-02-25T22:15:42.197 DEBUG:teuthology.orchestra.run.trial125:> sudo cp /boot/grub/grub.cfg /tmp/tmp.kMjPrtCKwF 2026-02-25T22:15:42.245 DEBUG:teuthology.orchestra.run.trial125:> sudo chmod 0666 /tmp/tmp.kMjPrtCKwF 2026-02-25T22:15:42.343 DEBUG:teuthology.orchestra.remote:trial125:/tmp/tmp.kMjPrtCKwF is 9KB 2026-02-25T22:15:42.351 DEBUG:teuthology.orchestra.run.trial125:> rm -fr /tmp/tmp.kMjPrtCKwF 2026-02-25T22:15:42.355 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2026-02-25T22:15:42.405 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-25T22:15:42.405 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/etc/grub.d/01_ceph_kernel 2026-02-25T22:15:42.405 DEBUG:teuthology.orchestra.run.trial125:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2026-02-25T22:15:42.459 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-171-generic 2026-02-25T22:15:42.459 DEBUG:teuthology.orchestra.run.trial125:> sudo update-grub 2026-02-25T22:15:42.689 INFO:teuthology.orchestra.run.trial125.stderr:Sourcing file `/etc/default/grub' 2026-02-25T22:15:42.709 INFO:teuthology.orchestra.run.trial125.stderr:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-25T22:15:42.710 INFO:teuthology.orchestra.run.trial125.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-25T22:15:42.712 INFO:teuthology.orchestra.run.trial125.stderr:Generating grub configuration file ... 2026-02-25T22:15:42.791 INFO:teuthology.orchestra.run.trial125.stderr:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-25T22:15:42.797 INFO:teuthology.orchestra.run.trial125.stderr:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-25T22:15:42.964 INFO:teuthology.orchestra.run.trial125.stderr:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-25T22:15:42.969 INFO:teuthology.orchestra.run.trial125.stderr:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-25T22:15:43.012 INFO:teuthology.orchestra.run.trial125.stderr:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-25T22:15:43.018 INFO:teuthology.orchestra.run.trial125.stderr:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-25T22:15:43.091 INFO:teuthology.orchestra.run.trial125.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-25T22:15:43.091 INFO:teuthology.orchestra.run.trial125.stderr:Systems on them will not be added to the GRUB boot configuration. 2026-02-25T22:15:43.091 INFO:teuthology.orchestra.run.trial125.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-25T22:15:43.094 INFO:teuthology.orchestra.run.trial125.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T22:15:43.101 INFO:teuthology.orchestra.run.trial125.stderr:done 2026-02-25T22:15:43.103 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial125 2026-02-25T22:15:43.103 DEBUG:teuthology.orchestra.run.trial125:> sudo bash -c ' 2026-02-25T22:15:43.103 DEBUG:teuthology.orchestra.run.trial125:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T22:15:43.103 DEBUG:teuthology.orchestra.run.trial125:> ' 2026-02-25T22:16:13.105 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T22:16:13.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial125.front.sepia.ceph.com' 2026-02-25T22:16:13.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T22:16:27.392 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.125 2026-02-25T22:16:36.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial125.front.sepia.ceph.com' 2026-02-25T22:16:36.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T22:17:12.443 DEBUG:teuthology.orchestra.run.trial125:> true 2026-02-25T22:17:12.837 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial125.front.sepia.ceph.com' 2026-02-25T22:17:12.837 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-171-generic"... 2026-02-25T22:17:12.837 DEBUG:teuthology.orchestra.run.trial125:> uname -r 2026-02-25T22:17:12.883 INFO:teuthology.orchestra.run.trial125.stdout:5.15.0-171-generic 2026-02-25T22:17:12.883 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-171-generic vs 5.15.0-171-generic 2026-02-25T22:17:12.883 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T22:17:12.883 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2026-02-25T22:17:13.884 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T22:17:13.884 DEBUG:teuthology.orchestra.run.trial125:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T22:17:13.897 INFO:teuthology.orchestra.run.trial125.stdout:ttyS1 2026-02-25T22:17:13.921 DEBUG:teuthology.parallel:result is None 2026-02-25T22:17:13.921 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T22:17:13.934 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T22:17:13.934 DEBUG:teuthology.orchestra.run.trial125:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T22:17:13.967 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T22:17:13.979 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T22:17:14.005 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T22:17:14.005 DEBUG:teuthology.orchestra.run.trial125:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T22:17:14.020 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T22:17:14.031 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T22:17:14.032 DEBUG:teuthology.orchestra.run.trial125:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T22:17:14.068 INFO:teuthology.orchestra.run.trial125.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:17:14.074 INFO:teuthology.orchestra.run.trial125.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:17:14.076 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T22:17:14.088 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T22:17:14.089 DEBUG:teuthology.orchestra.run.trial125:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T22:17:14.127 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T22:17:14.144 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T22:17:14.144 DEBUG:teuthology.orchestra.run.trial125:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T22:17:14.175 DEBUG:teuthology.orchestra.run.trial125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T22:17:14.219 DEBUG:teuthology.orchestra.run.trial125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T22:17:14.263 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-25T22:17:14.263 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T22:17:14.311 DEBUG:teuthology.orchestra.run.trial125:> sudo service rsyslog restart 2026-02-25T22:17:14.368 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T22:17:14.381 INFO:teuthology.task.internal:Starting timer... 2026-02-25T22:17:14.381 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T22:17:14.402 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T22:17:14.420 DEBUG:teuthology.task.selinux:Excluding trial125: OS 'ubuntu' does not support SELinux 2026-02-25T22:17:14.420 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T22:17:14.420 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-25T22:17:14.421 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T22:17:14.421 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T22:17:14.437 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-25T22:17:14.440 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-25T22:17:14.572 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T22:17:14.578 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T22:17:14.579 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial125.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T22:19:42.554 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial125.front.sepia.ceph.com')] 2026-02-25T22:19:42.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial125.front.sepia.ceph.com' 2026-02-25T22:19:42.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T22:19:42.663 DEBUG:teuthology.orchestra.run.trial125:> true 2026-02-25T22:19:42.911 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial125.front.sepia.ceph.com' 2026-02-25T22:19:42.911 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T22:19:42.931 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T22:19:42.932 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T22:19:42.932 DEBUG:teuthology.orchestra.run.trial125:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T22:19:42.975 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-25T22:19:42.975 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: Command line: ntpd -gq 2026-02-25T22:19:42.975 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: ---------------------------------------------------- 2026-02-25T22:19:42.975 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: ntp-4 is maintained by Network Time Foundation, 2026-02-25T22:19:42.975 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-25T22:19:42.975 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: corporation. Support and training for ntp-4 are 2026-02-25T22:19:42.975 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: available at https://www.nwtime.org/support 2026-02-25T22:19:42.975 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: ---------------------------------------------------- 2026-02-25T22:19:42.975 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: proto: precision = 0.030 usec (-25) 2026-02-25T22:19:42.976 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: basedate set to 2022-02-04 2026-02-25T22:19:42.976 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: gps base set to 2022-02-06 (week 2196) 2026-02-25T22:19:42.976 INFO:teuthology.orchestra.run.trial125.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-25T22:19:42.976 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-25T22:19:42.977 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: restrict ::: KOD does nothing without LIMITED. 2026-02-25T22:19:42.977 INFO:teuthology.orchestra.run.trial125.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-25T22:19:42.978 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: Listen and drop on 0 v6wildcard [::]:123 2026-02-25T22:19:42.978 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-25T22:19:42.978 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: Listen normally on 2 lo 127.0.0.1:123 2026-02-25T22:19:42.978 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: Listen normally on 3 enp1s0f0 10.20.193.125:123 2026-02-25T22:19:42.978 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: Listen normally on 4 lo [::1]:123 2026-02-25T22:19:42.978 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:61c2%2]:123 2026-02-25T22:19:42.978 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:43 ntpd[8836]: Listening on routing socket on fd #22 for interface updates 2026-02-25T22:19:45.975 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:45.975 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:45.975 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:45.975 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:45.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:45.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:45.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:45.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:45.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:45.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:45.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:45.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:45.984 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:45.984 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:45.984 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:45.984 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:46.002 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:46.002 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:46.002 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:46.002 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:46.014 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:46.014 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:46.014 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:46.015 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:46 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:47.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:47.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:47.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:47.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:47.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:47.977 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:47.977 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:47.977 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:47.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:47.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:47.979 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:47.979 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:47.984 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:47.984 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:47.984 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:47.984 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:48.001 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:48.002 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:48.002 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:48.002 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:48.014 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:48.014 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:48.014 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:48.014 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:48 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:49.975 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:49.975 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:49.975 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:49.975 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:49.975 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:49.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:49.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:49.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:49.977 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:49.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:49.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:49.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:49.983 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:49.984 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:49.984 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:49.984 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:50.001 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:50.001 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:50.002 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:50.002 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:50.014 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:50.014 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:50.014 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:50.014 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:50 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:51.975 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:51.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:51.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:51.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:51.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:51.976 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:51.977 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:51.977 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:51.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:19:51.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:19:51.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:19:51.978 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:52 ntpd[8836]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:19:51.978 INFO:teuthology.orchestra.run.trial125.stdout:25 Feb 22:19:51 ntpd[8836]: ntpd: time set -0.104603 s 2026-02-25T22:19:51.979 INFO:teuthology.orchestra.run.trial125.stdout:ntpd: time set -0.104603s 2026-02-25T22:19:51.979 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:51 ntpd[8836]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-25T22:19:51.979 INFO:teuthology.orchestra.run.trial125.stderr:25 Feb 22:19:51 ntpd[8836]: can't open /var/log/ntpstats/loopstats.20260225: Permission denied 2026-02-25T22:19:52.032 INFO:teuthology.orchestra.run.trial125.stdout: remote refid st t when poll reach delay offset jitter 2026-02-25T22:19:52.032 INFO:teuthology.orchestra.run.trial125.stdout:============================================================================== 2026-02-25T22:19:52.032 INFO:teuthology.orchestra.run.trial125.stdout: 172-232-15-202. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T22:19:52.033 INFO:teuthology.orchestra.run.trial125.stdout: 50.218.103.254 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T22:19:52.033 INFO:teuthology.orchestra.run.trial125.stdout: time-h.den.code .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T22:19:52.033 INFO:teuthology.orchestra.run.trial125.stdout: 172-104-28-175. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T22:19:52.034 INFO:teuthology.run_tasks:Running task install... 2026-02-25T22:19:52.048 DEBUG:teuthology.task.install:project ceph 2026-02-25T22:19:52.048 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd'}} 2026-02-25T22:19:52.048 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd'} 2026-02-25T22:19:52.048 INFO:teuthology.task.install:Using flavor: default 2026-02-25T22:19:52.057 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T22:19:52.057 INFO:teuthology.task.install:extra packages: [] 2026-02-25T22:19:52.058 DEBUG:teuthology.orchestra.run.trial125:> sudo apt-key list | grep Ceph 2026-02-25T22:19:52.133 INFO:teuthology.orchestra.run.trial125.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-25T22:19:52.162 INFO:teuthology.orchestra.run.trial125.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-25T22:19:52.163 INFO:teuthology.orchestra.run.trial125.stdout:uid [ unknown] Ceph.com (release key) 2026-02-25T22:19:52.164 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-osd, ceph-osd-classic, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-25T22:19:52.165 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T22:19:52.323 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default/ 2026-02-25T22:19:52.323 INFO:teuthology.task.install.deb:Package version is 20.3.0-5305-g0886691e-1jammy 2026-02-25T22:19:52.472 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-25T22:19:52.472 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-25T22:19:52.482 DEBUG:teuthology.orchestra.run.trial125:> sudo apt-get update 2026-02-25T22:19:52.639 INFO:teuthology.orchestra.run.trial125.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy InRelease 2026-02-25T22:19:52.651 INFO:teuthology.orchestra.run.trial125.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy Release [7689 B] 2026-02-25T22:19:52.663 INFO:teuthology.orchestra.run.trial125.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-25T22:19:52.675 INFO:teuthology.orchestra.run.trial125.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 Packages [19.2 kB] 2026-02-25T22:19:52.999 INFO:teuthology.orchestra.run.trial125.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T22:19:53.081 INFO:teuthology.orchestra.run.trial125.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-25T22:19:53.170 INFO:teuthology.orchestra.run.trial125.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-25T22:19:53.258 INFO:teuthology.orchestra.run.trial125.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-25T22:19:53.399 INFO:teuthology.orchestra.run.trial125.stdout:Fetched 26.9 kB in 1s (35.5 kB/s) 2026-02-25T22:19:53.927 INFO:teuthology.orchestra.run.trial125.stdout:Reading package lists... 2026-02-25T22:19:53.940 DEBUG:teuthology.orchestra.run.trial125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.3.0-5305-g0886691e-1jammy cephadm=20.3.0-5305-g0886691e-1jammy ceph-mds=20.3.0-5305-g0886691e-1jammy ceph-mgr=20.3.0-5305-g0886691e-1jammy ceph-osd=20.3.0-5305-g0886691e-1jammy ceph-osd-classic=20.3.0-5305-g0886691e-1jammy ceph-common=20.3.0-5305-g0886691e-1jammy ceph-fuse=20.3.0-5305-g0886691e-1jammy ceph-test=20.3.0-5305-g0886691e-1jammy ceph-volume=20.3.0-5305-g0886691e-1jammy radosgw=20.3.0-5305-g0886691e-1jammy python3-rados=20.3.0-5305-g0886691e-1jammy python3-rgw=20.3.0-5305-g0886691e-1jammy python3-cephfs=20.3.0-5305-g0886691e-1jammy python3-rbd=20.3.0-5305-g0886691e-1jammy libcephfs2=20.3.0-5305-g0886691e-1jammy libcephfs-dev=20.3.0-5305-g0886691e-1jammy librados2=20.3.0-5305-g0886691e-1jammy librbd1=20.3.0-5305-g0886691e-1jammy rbd-fuse=20.3.0-5305-g0886691e-1jammy 2026-02-25T22:19:53.973 INFO:teuthology.orchestra.run.trial125.stdout:Reading package lists... 2026-02-25T22:19:54.081 INFO:teuthology.orchestra.run.trial125.stdout:Building dependency tree... 2026-02-25T22:19:54.081 INFO:teuthology.orchestra.run.trial125.stdout:Reading state information... 2026-02-25T22:19:54.171 INFO:teuthology.orchestra.run.trial125.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T22:19:54.171 INFO:teuthology.orchestra.run.trial125.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T22:19:54.171 INFO:teuthology.orchestra.run.trial125.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout:The following additional packages will be installed: 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon jq libcephfs-daemon 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout: libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout: nvme-cli python-asyncssh-doc python3-asyncssh python3-cachetools 2026-02-25T22:19:54.172 INFO:teuthology.orchestra.run.trial125.stdout: python3-ceph-argparse python3-ceph-common python3-cheroot python3-cherrypy3 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: python3-google-auth python3-iniconfig python3-jaraco.classes 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: python3-joblib python3-kubernetes python3-natsort python3-pluggy 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: python3-portend python3-prettytable python3-psutil python3-py 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: python3-pygments python3-pytest python3-repoze.lru python3-requests-oauthlib 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: python3-routes python3-rsa python3-simplejson python3-sklearn 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: python3-sklearn-lib python3-tempora python3-threadpoolctl python3-toml 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: python3-wcwidth python3-webob python3-websocket python3-zc.lockfile 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: qttranslations5-l10n socat xmlstarlet 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout:Suggested packages: 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-25T22:19:54.173 INFO:teuthology.orchestra.run.trial125.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-25T22:19:54.174 INFO:teuthology.orchestra.run.trial125.stdout:Recommended packages: 2026-02-25T22:19:54.174 INFO:teuthology.orchestra.run.trial125.stdout: btrfs-tools 2026-02-25T22:19:54.205 INFO:teuthology.orchestra.run.trial125.stdout:The following NEW packages will be installed: 2026-02-25T22:19:54.206 INFO:teuthology.orchestra.run.trial125.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-25T22:19:54.206 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-25T22:19:54.206 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-osd-classic ceph-test 2026-02-25T22:19:54.206 INFO:teuthology.orchestra.run.trial125.stdout: ceph-volume cephadm jq libcephfs-daemon libcephfs-dev libcephfs-proxy2 2026-02-25T22:19:54.206 INFO:teuthology.orchestra.run.trial125.stdout: libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 2026-02-25T22:19:54.206 INFO:teuthology.orchestra.run.trial125.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-25T22:19:54.206 INFO:teuthology.orchestra.run.trial125.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-25T22:19:54.206 INFO:teuthology.orchestra.run.trial125.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-25T22:19:54.206 INFO:teuthology.orchestra.run.trial125.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-25T22:19:54.206 INFO:teuthology.orchestra.run.trial125.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-25T22:19:54.207 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-25T22:19:54.207 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-25T22:19:54.207 INFO:teuthology.orchestra.run.trial125.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-25T22:19:54.207 INFO:teuthology.orchestra.run.trial125.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-25T22:19:54.207 INFO:teuthology.orchestra.run.trial125.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-25T22:19:54.207 INFO:teuthology.orchestra.run.trial125.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-25T22:19:54.207 INFO:teuthology.orchestra.run.trial125.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-25T22:19:54.207 INFO:teuthology.orchestra.run.trial125.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-25T22:19:54.207 INFO:teuthology.orchestra.run.trial125.stdout: socat xmlstarlet 2026-02-25T22:19:54.285 INFO:teuthology.orchestra.run.trial125.stdout:0 upgraded, 87 newly installed, 0 to remove and 31 not upgraded. 2026-02-25T22:19:54.286 INFO:teuthology.orchestra.run.trial125.stdout:Need to get 702 MB of archives. 2026-02-25T22:19:54.286 INFO:teuthology.orchestra.run.trial125.stdout:After this operation, 2670 MB of additional disk space will be used. 2026-02-25T22:19:54.286 INFO:teuthology.orchestra.run.trial125.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.3.0-5305-g0886691e-1jammy [3797 kB] 2026-02-25T22:19:54.412 INFO:teuthology.orchestra.run.trial125.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.3.0-5305-g0886691e-1jammy [2910 kB] 2026-02-25T22:19:54.460 INFO:teuthology.orchestra.run.trial125.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.3.0-5305-g0886691e-1jammy [882 kB] 2026-02-25T22:19:54.471 INFO:teuthology.orchestra.run.trial125.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.3.0-5305-g0886691e-1jammy [359 kB] 2026-02-25T22:19:54.480 INFO:teuthology.orchestra.run.trial125.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.3.0-5305-g0886691e-1jammy [32.8 kB] 2026-02-25T22:19:54.481 INFO:teuthology.orchestra.run.trial125.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.3.0-5305-g0886691e-1jammy [225 kB] 2026-02-25T22:19:54.482 INFO:teuthology.orchestra.run.trial125.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.3.0-5305-g0886691e-1jammy [88.4 kB] 2026-02-25T22:19:54.482 INFO:teuthology.orchestra.run.trial125.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.3.0-5305-g0886691e-1jammy [338 kB] 2026-02-25T22:19:54.491 INFO:teuthology.orchestra.run.trial125.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.3.0-5305-g0886691e-1jammy [9092 kB] 2026-02-25T22:19:54.633 INFO:teuthology.orchestra.run.trial125.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-25T22:19:54.636 INFO:teuthology.orchestra.run.trial125.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.3.0-5305-g0886691e-1jammy [113 kB] 2026-02-25T22:19:54.637 INFO:teuthology.orchestra.run.trial125.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.3.0-5305-g0886691e-1jammy [261 kB] 2026-02-25T22:19:54.641 INFO:teuthology.orchestra.run.trial125.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.3.0-5305-g0886691e-1jammy [31.5 MB] 2026-02-25T22:19:55.063 INFO:teuthology.orchestra.run.trial125.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-25T22:19:55.077 INFO:teuthology.orchestra.run.trial125.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-25T22:19:55.162 INFO:teuthology.orchestra.run.trial125.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.3.0-5305-g0886691e-1jammy [4983 kB] 2026-02-25T22:19:55.166 INFO:teuthology.orchestra.run.trial125.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-25T22:19:55.260 INFO:teuthology.orchestra.run.trial125.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.3.0-5305-g0886691e-1jammy [293 kB] 2026-02-25T22:19:55.262 INFO:teuthology.orchestra.run.trial125.stdout:Get:19 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.3.0-5305-g0886691e-1jammy [127 kB] 2026-02-25T22:19:55.263 INFO:teuthology.orchestra.run.trial125.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.3.0-5305-g0886691e-1jammy [922 kB] 2026-02-25T22:19:55.267 INFO:teuthology.orchestra.run.trial125.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.3.0-5305-g0886691e-1jammy [7541 kB] 2026-02-25T22:19:55.388 INFO:teuthology.orchestra.run.trial125.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd-classic amd64 20.3.0-5305-g0886691e-1jammy [468 MB] 2026-02-25T22:19:55.421 INFO:teuthology.orchestra.run.trial125.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-25T22:19:55.434 INFO:teuthology.orchestra.run.trial125.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-25T22:19:55.468 INFO:teuthology.orchestra.run.trial125.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-25T22:19:55.478 INFO:teuthology.orchestra.run.trial125.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-25T22:19:55.481 INFO:teuthology.orchestra.run.trial125.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-25T22:19:55.482 INFO:teuthology.orchestra.run.trial125.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-25T22:19:55.482 INFO:teuthology.orchestra.run.trial125.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-25T22:19:55.504 INFO:teuthology.orchestra.run.trial125.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-25T22:19:55.505 INFO:teuthology.orchestra.run.trial125.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-25T22:19:55.512 INFO:teuthology.orchestra.run.trial125.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-25T22:19:55.597 INFO:teuthology.orchestra.run.trial125.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-25T22:19:55.597 INFO:teuthology.orchestra.run.trial125.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-25T22:19:55.597 INFO:teuthology.orchestra.run.trial125.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-25T22:19:55.598 INFO:teuthology.orchestra.run.trial125.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-25T22:19:55.598 INFO:teuthology.orchestra.run.trial125.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-25T22:19:55.598 INFO:teuthology.orchestra.run.trial125.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-25T22:19:55.598 INFO:teuthology.orchestra.run.trial125.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-25T22:19:55.604 INFO:teuthology.orchestra.run.trial125.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-25T22:19:55.604 INFO:teuthology.orchestra.run.trial125.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-25T22:19:55.692 INFO:teuthology.orchestra.run.trial125.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-25T22:19:55.695 INFO:teuthology.orchestra.run.trial125.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-25T22:19:55.695 INFO:teuthology.orchestra.run.trial125.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-25T22:19:55.696 INFO:teuthology.orchestra.run.trial125.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-25T22:19:55.728 INFO:teuthology.orchestra.run.trial125.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-25T22:19:55.730 INFO:teuthology.orchestra.run.trial125.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-25T22:19:55.730 INFO:teuthology.orchestra.run.trial125.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-25T22:19:55.804 INFO:teuthology.orchestra.run.trial125.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-25T22:19:55.804 INFO:teuthology.orchestra.run.trial125.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-25T22:19:55.804 INFO:teuthology.orchestra.run.trial125.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-25T22:19:55.805 INFO:teuthology.orchestra.run.trial125.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-25T22:19:55.805 INFO:teuthology.orchestra.run.trial125.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-25T22:19:55.805 INFO:teuthology.orchestra.run.trial125.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-25T22:19:55.881 INFO:teuthology.orchestra.run.trial125.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-25T22:19:55.882 INFO:teuthology.orchestra.run.trial125.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-25T22:19:55.884 INFO:teuthology.orchestra.run.trial125.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-25T22:19:55.972 INFO:teuthology.orchestra.run.trial125.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-25T22:19:55.976 INFO:teuthology.orchestra.run.trial125.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-25T22:19:55.978 INFO:teuthology.orchestra.run.trial125.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-25T22:19:55.983 INFO:teuthology.orchestra.run.trial125.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-25T22:19:55.987 INFO:teuthology.orchestra.run.trial125.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-25T22:19:55.987 INFO:teuthology.orchestra.run.trial125.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-25T22:19:55.987 INFO:teuthology.orchestra.run.trial125.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-25T22:19:56.064 INFO:teuthology.orchestra.run.trial125.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-25T22:19:56.064 INFO:teuthology.orchestra.run.trial125.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-25T22:19:56.072 INFO:teuthology.orchestra.run.trial125.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-25T22:19:56.155 INFO:teuthology.orchestra.run.trial125.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-25T22:19:56.158 INFO:teuthology.orchestra.run.trial125.stdout:Get:69 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-25T22:19:56.158 INFO:teuthology.orchestra.run.trial125.stdout:Get:70 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-25T22:19:56.159 INFO:teuthology.orchestra.run.trial125.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-25T22:20:13.764 INFO:teuthology.orchestra.run.trial125.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.3.0-5305-g0886691e-1jammy [180 kB] 2026-02-25T22:20:13.765 INFO:teuthology.orchestra.run.trial125.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.3.0-5305-g0886691e-1jammy [14.0 kB] 2026-02-25T22:20:13.765 INFO:teuthology.orchestra.run.trial125.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.3.0-5305-g0886691e-1jammy [1030 kB] 2026-02-25T22:20:13.770 INFO:teuthology.orchestra.run.trial125.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.3.0-5305-g0886691e-1jammy [2437 kB] 2026-02-25T22:20:13.783 INFO:teuthology.orchestra.run.trial125.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.3.0-5305-g0886691e-1jammy [1069 kB] 2026-02-25T22:20:13.816 INFO:teuthology.orchestra.run.trial125.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.3.0-5305-g0886691e-1jammy [190 kB] 2026-02-25T22:20:13.823 INFO:teuthology.orchestra.run.trial125.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.3.0-5305-g0886691e-1jammy [54.7 MB] 2026-02-25T22:20:16.566 INFO:teuthology.orchestra.run.trial125.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.3.0-5305-g0886691e-1jammy [8626 kB] 2026-02-25T22:20:17.021 INFO:teuthology.orchestra.run.trial125.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.3.0-5305-g0886691e-1jammy [14.2 kB] 2026-02-25T22:20:17.021 INFO:teuthology.orchestra.run.trial125.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.3.0-5305-g0886691e-1jammy [68.6 MB] 2026-02-25T22:20:20.943 INFO:teuthology.orchestra.run.trial125.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.3.0-5305-g0886691e-1jammy [137 kB] 2026-02-25T22:20:20.944 INFO:teuthology.orchestra.run.trial125.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.3.0-5305-g0886691e-1jammy [45.8 kB] 2026-02-25T22:20:20.944 INFO:teuthology.orchestra.run.trial125.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.3.0-5305-g0886691e-1jammy [32.4 kB] 2026-02-25T22:20:20.945 INFO:teuthology.orchestra.run.trial125.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.3.0-5305-g0886691e-1jammy [44.2 kB] 2026-02-25T22:20:20.945 INFO:teuthology.orchestra.run.trial125.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.3.0-5305-g0886691e-1jammy [18.3 MB] 2026-02-25T22:20:22.056 INFO:teuthology.orchestra.run.trial125.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.3.0-5305-g0886691e-1jammy [94.1 kB] 2026-02-25T22:20:22.335 INFO:teuthology.orchestra.run.trial125.stdout:Fetched 702 MB in 28s (25.2 MB/s) 2026-02-25T22:20:22.351 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-25T22:20:22.519 INFO:teuthology.orchestra.run.trial125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 202926 files and directories currently installed.) 2026-02-25T22:20:22.522 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-25T22:20:22.523 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-25T22:20:22.541 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-25T22:20:22.555 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-25T22:20:22.555 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-25T22:20:22.572 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-25T22:20:22.584 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-25T22:20:22.585 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-25T22:20:22.602 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-25T22:20:22.613 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T22:20:22.618 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:20:22.649 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-25T22:20:22.659 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T22:20:22.660 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:20:22.676 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-25T22:20:22.686 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T22:20:22.687 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:20:22.706 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-25T22:20:22.716 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-25T22:20:22.716 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-25T22:20:22.733 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package librados2. 2026-02-25T22:20:22.743 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../07-librados2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:22.743 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking librados2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:22.787 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libnbd0. 2026-02-25T22:20:22.798 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-25T22:20:22.798 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-25T22:20:22.813 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package librbd1. 2026-02-25T22:20:22.823 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../09-librbd1_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:22.824 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking librbd1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:22.862 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libcephfs2. 2026-02-25T22:20:22.871 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../10-libcephfs2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:22.872 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libcephfs2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:22.893 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-rados. 2026-02-25T22:20:22.902 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../11-python3-rados_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:22.903 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-rados (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:22.920 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-25T22:20:22.931 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:20:22.931 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-ceph-argparse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:22.947 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-cephfs. 2026-02-25T22:20:22.956 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../13-python3-cephfs_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:22.957 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-cephfs (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:22.973 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-25T22:20:22.983 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../14-python3-ceph-common_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:20:22.984 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:23.002 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-25T22:20:23.012 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-25T22:20:23.012 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-25T22:20:23.029 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-prettytable. 2026-02-25T22:20:23.039 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-25T22:20:23.039 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-25T22:20:23.054 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-rbd. 2026-02-25T22:20:23.063 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../17-python3-rbd_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:23.064 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-rbd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:23.082 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-25T22:20:23.092 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-25T22:20:23.092 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-25T22:20:23.112 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package librgw2. 2026-02-25T22:20:23.122 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../19-librgw2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:23.122 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking librgw2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:23.220 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-rgw. 2026-02-25T22:20:23.230 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../20-python3-rgw_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:23.231 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-rgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:23.247 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-25T22:20:23.256 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-25T22:20:23.257 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-25T22:20:23.272 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libradosstriper1. 2026-02-25T22:20:23.282 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../22-libradosstriper1_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:23.282 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libradosstriper1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:23.299 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-common. 2026-02-25T22:20:23.309 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../23-ceph-common_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:23.310 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:23.635 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-base. 2026-02-25T22:20:23.645 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../24-ceph-base_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:23.650 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-base (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:23.711 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-25T22:20:23.721 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-25T22:20:23.721 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-25T22:20:23.736 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-cheroot. 2026-02-25T22:20:23.746 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-25T22:20:23.747 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-25T22:20:23.764 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-25T22:20:23.774 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-25T22:20:23.775 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-25T22:20:23.789 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-25T22:20:23.799 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-25T22:20:23.799 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-25T22:20:23.814 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-25T22:20:23.825 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-25T22:20:23.825 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-25T22:20:23.840 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-tempora. 2026-02-25T22:20:23.850 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-25T22:20:23.851 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-25T22:20:23.865 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-portend. 2026-02-25T22:20:23.876 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-25T22:20:23.876 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-25T22:20:23.891 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-25T22:20:23.901 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-25T22:20:23.901 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-25T22:20:23.917 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-25T22:20:23.927 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-25T22:20:23.927 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-25T22:20:23.952 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-natsort. 2026-02-25T22:20:23.962 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-25T22:20:23.962 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-25T22:20:23.978 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-25T22:20:23.988 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:20:23.989 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-mgr-modules-core (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:24.020 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-25T22:20:24.030 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:24.031 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libsqlite3-mod-ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:24.047 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-mgr. 2026-02-25T22:20:24.057 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../37-ceph-mgr_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:24.058 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-mgr (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:24.080 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-mon. 2026-02-25T22:20:24.090 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../38-ceph-mon_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:24.090 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-mon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:24.155 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-25T22:20:24.166 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-25T22:20:24.166 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-25T22:20:24.182 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-osd-classic. 2026-02-25T22:20:24.192 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../40-ceph-osd-classic_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:24.193 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-osd-classic (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:28.257 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-osd. 2026-02-25T22:20:28.273 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../41-ceph-osd_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:28.273 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-osd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:28.293 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph. 2026-02-25T22:20:28.308 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../42-ceph_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:28.308 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:28.325 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-fuse. 2026-02-25T22:20:28.335 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../43-ceph-fuse_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:28.336 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:28.358 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-mds. 2026-02-25T22:20:28.368 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../44-ceph-mds_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:28.369 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-mds (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:28.402 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package cephadm. 2026-02-25T22:20:28.412 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../45-cephadm_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:28.412 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:28.429 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-25T22:20:28.439 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../46-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-25T22:20:28.439 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-25T22:20:28.461 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-25T22:20:28.472 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../47-ceph-mgr-cephadm_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:20:28.472 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-mgr-cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:28.494 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-25T22:20:28.504 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../48-python3-repoze.lru_0.7-2_all.deb ... 2026-02-25T22:20:28.504 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-25T22:20:28.521 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-routes. 2026-02-25T22:20:28.531 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../49-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-25T22:20:28.531 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-25T22:20:28.552 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-25T22:20:28.562 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../50-ceph-mgr-dashboard_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:20:28.562 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-mgr-dashboard (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:29.238 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-25T22:20:29.248 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../51-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-25T22:20:29.248 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-25T22:20:29.287 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-joblib. 2026-02-25T22:20:29.298 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../52-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-25T22:20:29.299 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-25T22:20:29.326 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-25T22:20:29.336 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../53-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-25T22:20:29.337 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-25T22:20:29.352 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-sklearn. 2026-02-25T22:20:29.363 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../54-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-25T22:20:29.364 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-25T22:20:29.447 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-25T22:20:29.458 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../55-ceph-mgr-diskprediction-local_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:20:29.458 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-mgr-diskprediction-local (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:29.619 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-cachetools. 2026-02-25T22:20:29.629 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../56-python3-cachetools_5.0.0-1_all.deb ... 2026-02-25T22:20:29.629 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-25T22:20:29.645 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-rsa. 2026-02-25T22:20:29.656 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../57-python3-rsa_4.8-1_all.deb ... 2026-02-25T22:20:29.656 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-25T22:20:29.674 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-google-auth. 2026-02-25T22:20:29.685 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../58-python3-google-auth_1.5.1-3_all.deb ... 2026-02-25T22:20:29.685 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-25T22:20:29.703 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-25T22:20:29.714 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../59-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-25T22:20:29.714 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-25T22:20:29.730 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-websocket. 2026-02-25T22:20:29.740 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../60-python3-websocket_1.2.3-1_all.deb ... 2026-02-25T22:20:29.740 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-25T22:20:29.759 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-25T22:20:29.769 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../61-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-25T22:20:29.770 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-25T22:20:29.865 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-25T22:20:29.876 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../62-ceph-mgr-k8sevents_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:20:29.876 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-mgr-k8sevents (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:29.891 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-25T22:20:29.902 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../63-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-25T22:20:29.902 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-25T22:20:29.918 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-25T22:20:29.929 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../64-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-25T22:20:29.929 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-25T22:20:29.944 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package jq. 2026-02-25T22:20:29.954 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../65-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-25T22:20:29.955 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-25T22:20:29.970 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package socat. 2026-02-25T22:20:29.980 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../66-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-25T22:20:29.981 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-25T22:20:30.001 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package xmlstarlet. 2026-02-25T22:20:30.010 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../67-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-25T22:20:30.010 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-25T22:20:30.047 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-test. 2026-02-25T22:20:30.057 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../68-ceph-test_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:30.057 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-test (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:30.695 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package ceph-volume. 2026-02-25T22:20:30.706 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../69-ceph-volume_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:20:30.706 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking ceph-volume (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:30.729 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-25T22:20:30.739 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../70-libcephfs-daemon_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:30.739 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libcephfs-daemon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:30.756 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-25T22:20:30.766 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../71-libcephfs-proxy2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:30.766 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libcephfs-proxy2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:30.781 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-25T22:20:30.791 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../72-libcephfs-dev_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:30.791 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking libcephfs-dev (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:30.807 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package nvme-cli. 2026-02-25T22:20:30.818 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../73-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-25T22:20:30.818 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-25T22:20:30.852 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-25T22:20:30.862 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../74-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-25T22:20:30.863 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-25T22:20:30.893 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-25T22:20:30.903 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../75-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-25T22:20:30.904 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-25T22:20:30.919 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-pluggy. 2026-02-25T22:20:30.929 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../76-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-25T22:20:30.929 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-25T22:20:30.946 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-psutil. 2026-02-25T22:20:30.957 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../77-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-25T22:20:30.957 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-25T22:20:30.976 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-py. 2026-02-25T22:20:30.986 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../78-python3-py_1.10.0-1_all.deb ... 2026-02-25T22:20:30.991 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-25T22:20:31.008 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-pygments. 2026-02-25T22:20:31.018 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../79-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-25T22:20:31.019 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-25T22:20:31.060 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-toml. 2026-02-25T22:20:31.071 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../80-python3-toml_0.10.2-1_all.deb ... 2026-02-25T22:20:31.071 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-25T22:20:31.087 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-pytest. 2026-02-25T22:20:31.097 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../81-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-25T22:20:31.097 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-25T22:20:31.120 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-simplejson. 2026-02-25T22:20:31.130 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../82-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-25T22:20:31.131 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-25T22:20:31.156 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package python3-webob. 2026-02-25T22:20:31.166 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../83-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-25T22:20:31.167 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-25T22:20:31.184 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-25T22:20:31.194 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../84-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-25T22:20:31.195 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-25T22:20:31.264 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package radosgw. 2026-02-25T22:20:31.275 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../85-radosgw_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:31.275 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking radosgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:31.435 INFO:teuthology.orchestra.run.trial125.stdout:Selecting previously unselected package rbd-fuse. 2026-02-25T22:20:31.446 INFO:teuthology.orchestra.run.trial125.stdout:Preparing to unpack .../86-rbd-fuse_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:20:31.446 INFO:teuthology.orchestra.run.trial125.stdout:Unpacking rbd-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:31.474 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-25T22:20:31.539 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-25T22:20:31.540 INFO:teuthology.orchestra.run.trial125.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-25T22:20:31.612 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T22:20:31.875 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-25T22:20:32.253 INFO:teuthology.orchestra.run.trial125.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-25T22:20:32.264 INFO:teuthology.orchestra.run.trial125.stdout:Setting up cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:32.306 INFO:teuthology.orchestra.run.trial125.stdout:Adding system user cephadm....done 2026-02-25T22:20:32.311 INFO:teuthology.orchestra.run.trial125.stdout:usermod: no changes 2026-02-25T22:20:32.317 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-25T22:20:32.388 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-25T22:20:32.389 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-25T22:20:32.460 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-25T22:20:32.535 INFO:teuthology.orchestra.run.trial125.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-25T22:20:32.536 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-25T22:20:32.630 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-25T22:20:32.754 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-25T22:20:32.827 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-25T22:20:32.901 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-ceph-argparse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:32.966 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-25T22:20:32.967 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-25T22:20:32.968 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-25T22:20:32.968 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-25T22:20:32.969 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-25T22:20:33.089 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-25T22:20:33.165 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libcephfs-proxy2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:33.166 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-25T22:20:33.242 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-25T22:20:33.332 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-25T22:20:33.577 INFO:teuthology.orchestra.run.trial125.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-25T22:20:33.578 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-25T22:20:33.674 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-25T22:20:33.799 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-25T22:20:33.881 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-25T22:20:33.952 INFO:teuthology.orchestra.run.trial125.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-25T22:20:33.953 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:34.041 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-25T22:20:34.531 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:20:34.536 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-25T22:20:34.612 INFO:teuthology.orchestra.run.trial125.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-25T22:20:34.613 INFO:teuthology.orchestra.run.trial125.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-25T22:20:34.614 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-25T22:20:34.690 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-25T22:20:34.764 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:20:34.765 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-25T22:20:34.842 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-25T22:20:34.918 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-25T22:20:34.999 INFO:teuthology.orchestra.run.trial125.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-25T22:20:35.000 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-25T22:20:35.082 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-25T22:20:35.083 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-25T22:20:35.160 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-25T22:20:35.242 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-25T22:20:35.317 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-25T22:20:35.318 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-25T22:20:35.449 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-25T22:20:35.514 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:20:35.515 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-25T22:20:35.596 INFO:teuthology.orchestra.run.trial125.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-25T22:20:35.597 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-25T22:20:35.726 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-25T22:20:35.727 INFO:teuthology.orchestra.run.trial125.stdout:Setting up librados2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:35.728 INFO:teuthology.orchestra.run.trial125.stdout:Setting up librgw2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:35.729 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libsqlite3-mod-ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:35.730 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-25T22:20:36.247 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libcephfs2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.248 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libradosstriper1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.249 INFO:teuthology.orchestra.run.trial125.stdout:Setting up librbd1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.250 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-mgr-modules-core (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.251 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.321 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-25T22:20:36.322 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-25T22:20:36.697 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libcephfs-dev (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.698 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-rados (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.699 INFO:teuthology.orchestra.run.trial125.stdout:Setting up libcephfs-daemon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.700 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-rbd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.701 INFO:teuthology.orchestra.run.trial125.stdout:Setting up rbd-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.702 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-rgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.703 INFO:teuthology.orchestra.run.trial125.stdout:Setting up python3-cephfs (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.704 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:36.732 INFO:teuthology.orchestra.run.trial125.stdout:Adding group ceph....done 2026-02-25T22:20:36.767 INFO:teuthology.orchestra.run.trial125.stdout:Adding system user ceph....done 2026-02-25T22:20:36.778 INFO:teuthology.orchestra.run.trial125.stdout:Setting system user ceph properties....done 2026-02-25T22:20:36.784 INFO:teuthology.orchestra.run.trial125.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-25T22:20:36.871 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-25T22:20:37.133 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-25T22:20:37.508 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-test (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:37.509 INFO:teuthology.orchestra.run.trial125.stdout:Setting up radosgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:37.750 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-25T22:20:37.750 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-25T22:20:38.108 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-base (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:38.225 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-25T22:20:38.582 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-mds (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:38.644 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-25T22:20:38.644 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-25T22:20:38.983 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-mgr (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:39.045 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-25T22:20:39.046 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-25T22:20:39.408 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-mgr-k8sevents (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:39.409 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-osd-classic (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:39.425 INFO:teuthology.orchestra.run.trial125.stdout:update-alternatives: using /usr/bin/ceph-osd-classic to provide /usr/bin/ceph-osd (ceph-osd) in auto mode 2026-02-25T22:20:39.435 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-mgr-diskprediction-local (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:39.445 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-mon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:39.513 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-25T22:20:39.513 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-25T22:20:39.872 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-mgr-cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:39.883 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-mgr-dashboard (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:39.894 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-osd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:39.967 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-25T22:20:39.967 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-25T22:20:40.334 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:40.335 INFO:teuthology.orchestra.run.trial125.stdout:Setting up ceph-volume (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:20:40.442 INFO:teuthology.orchestra.run.trial125.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-25T22:20:40.464 INFO:teuthology.orchestra.run.trial125.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-25T22:20:41.107 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:20:41.107 INFO:teuthology.orchestra.run.trial125.stdout:Running kernel seems to be up-to-date. 2026-02-25T22:20:41.107 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:20:41.107 INFO:teuthology.orchestra.run.trial125.stdout:The processor microcode seems to be up-to-date. 2026-02-25T22:20:41.107 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:20:41.107 INFO:teuthology.orchestra.run.trial125.stdout:No services need to be restarted. 2026-02-25T22:20:41.107 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:20:41.107 INFO:teuthology.orchestra.run.trial125.stdout:No containers need to be restarted. 2026-02-25T22:20:41.107 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:20:41.108 INFO:teuthology.orchestra.run.trial125.stdout:No user sessions are running outdated binaries. 2026-02-25T22:20:41.108 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-25T22:20:41.108 INFO:teuthology.orchestra.run.trial125.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-25T22:20:41.908 INFO:teuthology.orchestra.run.trial125.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-25T22:20:41.917 DEBUG:teuthology.parallel:result is None 2026-02-25T22:20:41.918 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T22:20:47.096 DEBUG:teuthology.orchestra.run.trial125:> dpkg-query -W -f '${Version}' ceph 2026-02-25T22:20:47.108 INFO:teuthology.orchestra.run.trial125.stdout:20.3.0-5305-g0886691e-1jammy 2026-02-25T22:20:47.109 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5305-g0886691e-1jammy 2026-02-25T22:20:47.109 INFO:teuthology.task.install:The correct ceph version 20.3.0-5305-g0886691e-1jammy is installed. 2026-02-25T22:20:47.111 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T22:20:47.111 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-25T22:20:47.111 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T22:20:47.160 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T22:20:47.160 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-25T22:20:47.161 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T22:20:47.212 DEBUG:teuthology.orchestra.run.trial125:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T22:20:47.263 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T22:20:47.264 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-25T22:20:47.264 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T22:20:47.316 DEBUG:teuthology.orchestra.run.trial125:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T22:20:47.366 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T22:20:47.367 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-25T22:20:47.367 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T22:20:47.416 DEBUG:teuthology.orchestra.run.trial125:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T22:20:47.467 INFO:teuthology.run_tasks:Running task workunit... 2026-02-25T22:20:47.495 INFO:tasks.workunit:Pulling workunits from ref 61d4431d3ec879bd98659388b3655acf3d392418 2026-02-25T22:20:47.496 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-25T22:20:47.496 INFO:tasks.workunit:timeout=3h 2026-02-25T22:20:47.496 INFO:tasks.workunit:cleanup=True 2026-02-25T22:20:47.497 DEBUG:teuthology.orchestra.run.trial125:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-25T22:20:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T22:20:47.511 INFO:teuthology.orchestra.run.trial125.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-25T22:20:47.511 DEBUG:teuthology.orchestra.run.trial125:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T22:20:47.558 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-25T22:20:47.559 DEBUG:teuthology.orchestra.run.trial125:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-25T22:20:47.603 DEBUG:teuthology.orchestra.run.trial125:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/Ericmzhang/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 61d4431d3ec879bd98659388b3655acf3d392418 2026-02-25T22:20:47.653 INFO:tasks.workunit.client.0.trial125.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-25T22:21:05.413 INFO:tasks.workunit.client.0.trial125.stderr:Note: switching to '61d4431d3ec879bd98659388b3655acf3d392418'. 2026-02-25T22:21:05.413 INFO:tasks.workunit.client.0.trial125.stderr: 2026-02-25T22:21:05.413 INFO:tasks.workunit.client.0.trial125.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-25T22:21:05.413 INFO:tasks.workunit.client.0.trial125.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-25T22:21:05.413 INFO:tasks.workunit.client.0.trial125.stderr:state without impacting any branches by switching back to a branch. 2026-02-25T22:21:05.413 INFO:tasks.workunit.client.0.trial125.stderr: 2026-02-25T22:21:05.413 INFO:tasks.workunit.client.0.trial125.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-25T22:21:05.413 INFO:tasks.workunit.client.0.trial125.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-25T22:21:05.414 INFO:tasks.workunit.client.0.trial125.stderr: 2026-02-25T22:21:05.414 INFO:tasks.workunit.client.0.trial125.stderr: git switch -c 2026-02-25T22:21:05.414 INFO:tasks.workunit.client.0.trial125.stderr: 2026-02-25T22:21:05.414 INFO:tasks.workunit.client.0.trial125.stderr:Or undo this operation with: 2026-02-25T22:21:05.414 INFO:tasks.workunit.client.0.trial125.stderr: 2026-02-25T22:21:05.414 INFO:tasks.workunit.client.0.trial125.stderr: git switch - 2026-02-25T22:21:05.414 INFO:tasks.workunit.client.0.trial125.stderr: 2026-02-25T22:21:05.414 INFO:tasks.workunit.client.0.trial125.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-25T22:21:05.414 INFO:tasks.workunit.client.0.trial125.stderr: 2026-02-25T22:21:05.414 INFO:tasks.workunit.client.0.trial125.stderr:HEAD is now at 61d4431d3ec changed test script name so it runs first 2026-02-25T22:21:05.416 DEBUG:teuthology.orchestra.run.trial125:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-25T22:21:05.469 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-25T22:21:05.469 DEBUG:teuthology.orchestra.run.trial125:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-25T22:21:05.516 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2026-02-25T22:21:05.517 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2026-02-25T22:21:05.517 DEBUG:teuthology.orchestra.run.trial125:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61d4431d3ec879bd98659388b3655acf3d392418 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2026-02-25T22:21:05.566 INFO:tasks.workunit.client.0.trial125.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2026-02-25T22:21:05.567 INFO:tasks.workunit.client.0.trial125.stderr:++ TIMEOUT=300 2026-02-25T22:21:05.567 INFO:tasks.workunit.client.0.trial125.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2026-02-25T22:21:05.567 INFO:tasks.workunit.client.0.trial125.stderr:++ MAX_TIMEOUT=15 2026-02-25T22:21:05.567 INFO:tasks.workunit.client.0.trial125.stderr:++ PG_NUM=4 2026-02-25T22:21:05.567 INFO:tasks.workunit.client.0.trial125.stderr:++ TMPDIR=/tmp 2026-02-25T22:21:05.567 INFO:tasks.workunit.client.0.trial125.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2026-02-25T22:21:05.567 INFO:tasks.workunit.client.0.trial125.stderr:++ TESTDIR=/home/ubuntu/cephtest 2026-02-25T22:21:05.567 INFO:tasks.workunit.client.0.trial125.stderr:+++ uname 2026-02-25T22:21:05.568 INFO:tasks.workunit.client.0.trial125.stderr:++ '[' Linux = FreeBSD ']' 2026-02-25T22:21:05.568 INFO:tasks.workunit.client.0.trial125.stderr:++ SED=sed 2026-02-25T22:21:05.569 INFO:tasks.workunit.client.0.trial125.stderr:++ AWK=awk 2026-02-25T22:21:05.569 INFO:tasks.workunit.client.0.trial125.stderr:+++ stty -a 2026-02-25T22:21:05.569 INFO:tasks.workunit.client.0.trial125.stderr:+++ head -1 2026-02-25T22:21:05.569 INFO:tasks.workunit.client.0.trial125.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2026-02-25T22:21:05.570 INFO:tasks.workunit.client.0.trial125.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T22:21:05.570 INFO:tasks.workunit.client.0.trial125.stderr:++ termwidth= 2026-02-25T22:21:05.570 INFO:tasks.workunit.client.0.trial125.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2026-02-25T22:21:05.570 INFO:tasks.workunit.client.0.trial125.stderr:++ DIFFCOLOPTS='-y ' 2026-02-25T22:21:05.570 INFO:tasks.workunit.client.0.trial125.stderr:++ KERNCORE=kernel.core_pattern 2026-02-25T22:21:05.570 INFO:tasks.workunit.client.0.trial125.stderr:++ EXTRA_OPTS= 2026-02-25T22:21:05.571 INFO:tasks.workunit.client.0.trial125.stderr:++ test '' = TESTS 2026-02-25T22:21:05.571 INFO:tasks.workunit.client.0.trial125.stderr:++ uname -m 2026-02-25T22:21:05.574 INFO:tasks.workunit.client.0.trial125.stderr:+ arch=x86_64 2026-02-25T22:21:05.574 INFO:tasks.workunit.client.0.trial125.stderr:+ case $arch in 2026-02-25T22:21:05.574 INFO:tasks.workunit.client.0.trial125.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2026-02-25T22:21:05.574 INFO:tasks.workunit.client.0.trial125.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2026-02-25T22:21:05.574 INFO:tasks.workunit.client.0.trial125.stderr:+ plugins=(jerasure shec lrc isa) 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:+ main test-erasure-code-plugins 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:+ local dir=td/test-erasure-code-plugins 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:+ shift 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:+ shopt -s -o xtrace 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T22:21:05.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T22:21:05.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T22:21:05.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/test-erasure-code-plugins 2026-02-25T22:21:05.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:05.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2026-02-25T22:21:05.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2026-02-25T22:21:05.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2026-02-25T22:21:05.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2026-02-25T22:21:05.576 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2026-02-25T22:21:05.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none ' 2026-02-25T22:21:05.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2026-02-25T22:21:05.578 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2026-02-25T22:21:05.578 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T22:21:05.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_blaum_roth_warning 2026-02-25T22:21:05.579 INFO:tasks.workunit.client.0.trial125.stderr:TEST_ec_profile_warning 2026-02-25T22:21:05.579 INFO:tasks.workunit.client.0.trial125.stderr:TEST_preload_no_warning 2026-02-25T22:21:05.579 INFO:tasks.workunit.client.0.trial125.stderr:TEST_preload_no_warning_default 2026-02-25T22:21:05.579 INFO:tasks.workunit.client.0.trial125.stderr:TEST_preload_warning' 2026-02-25T22:21:05.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-25T22:21:05.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_blaum_roth_warning td/test-erasure-code-plugins 2026-02-25T22:21:05.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:119: TEST_ec_profile_blaum_roth_warning: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:05.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:121: TEST_ec_profile_blaum_roth_warning: setup td/test-erasure-code-plugins 2026-02-25T22:21:05.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:05.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:21:05.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:05.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:05.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:21:05.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:05.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:05.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:05.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:05.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:05.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:05.581 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:05.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:05.582 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:05.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:05.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:05.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:05.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:05.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:05.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:05.585 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:05.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:05.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:05.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:21:05.589 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:05.589 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:05.589 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:05.589 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:21:05.590 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:05.590 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:05.590 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:21:05.592 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:21:05.592 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:05.592 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:05.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:21:05.593 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:21:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T22:21:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T22:21:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:21:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:21:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:122: TEST_ec_profile_blaum_roth_warning: run_mon td/test-erasure-code-plugins a 2026-02-25T22:21:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:21:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:21:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:21:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:21:05.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2026-02-25T22:21:05.636 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:21:05.636 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:05.636 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:05.637 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:05.637 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:05.637 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:05.638 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:05.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:21:05.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:21:05.671 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:21:05.671 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:05.671 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:05.671 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:21:05.673 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:21:05.673 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:05.673 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:05.673 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:05.673 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:05.673 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:05.673 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:05.673 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:05.673 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:05.673 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:21:05.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:21:05.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:05.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:05.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:21:05.728 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:21:05.729 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:05.729 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:05.729 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:05.729 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:05.729 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:05.729 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:05.729 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:05.730 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:05.730 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:21:05.784 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:123: TEST_ec_profile_blaum_roth_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:21:05.784 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:05.784 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:21:05.784 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:21:05.784 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:21:05.784 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:21:05.784 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:21:05.895 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:21:05.895 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:05.895 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:05.895 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:05.895 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:05.895 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:05.895 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:05.896 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:05.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:05.914 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:124: TEST_ec_profile_blaum_roth_warning: seq 0 2 2026-02-25T22:21:05.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:124: TEST_ec_profile_blaum_roth_warning: for id in $(seq 0 2) 2026-02-25T22:21:05.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:125: TEST_ec_profile_blaum_roth_warning: run_osd td/test-erasure-code-plugins 0 2026-02-25T22:21:05.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:05.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:21:05.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:21:05.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:21:05.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:21:05.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:05.919 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:05.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:05.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:21:05.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:21:05.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:21:05.921 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:21:05.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e1c6458e-4e37-4fc7-9862-0eeeb480d618 2026-02-25T22:21:05.922 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e1c6458e-4e37-4fc7-9862-0eeeb480d618' 2026-02-25T22:21:05.922 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 e1c6458e-4e37-4fc7-9862-0eeeb480d618 2026-02-25T22:21:05.922 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:21:05.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDRdZ9p69LBNxAAo0BzpZeai9P4nuEi1KaBWw== 2026-02-25T22:21:05.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDRdZ9p69LBNxAAo0BzpZeai9P4nuEi1KaBWw=="}' 2026-02-25T22:21:05.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e1c6458e-4e37-4fc7-9862-0eeeb480d618 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:21:06.040 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:06.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:21:06.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRdZ9p69LBNxAAo0BzpZeai9P4nuEi1KaBWw== --osd-uuid e1c6458e-4e37-4fc7-9862-0eeeb480d618 2026-02-25T22:21:06.062 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:06.059+0000 7efde0d0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:06.064 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:06.063+0000 7efde0d0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:06.065 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:06.063+0000 7efde0d0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:06.065 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:06.063+0000 7efde0d0aa40 -1 bdev(0x55b0ea2c6400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:21:06.066 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:06.063+0000 7efde0d0aa40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:21:06.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:21:06.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:21:06.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:21:06.771 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:21:06.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:21:06.889 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:21:06.889 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:21:06.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:21:06.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:21:06.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:21:06.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:21:06.911 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:06.907+0000 7fe744ad1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:06.912 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:06.911+0000 7fe744ad1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:06.914 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:06.911+0000 7fe744ad1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:06.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:21:06.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:06.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:21:06.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:06.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:06.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:06.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:06.999 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:06.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:06.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:07.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:07.477 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:07.475+0000 7fe744ad1a40 -1 Falling back to public interface 2026-02-25T22:21:08.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:08.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:08.104 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:21:08.105 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:08.105 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:08.105 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:08.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:08.486 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:08.483+0000 7fe744ad1a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:21:09.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:09.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:09.209 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:21:09.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:21:09.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:09.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:09.420 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2331858124,v1:127.0.0.1:6801/2331858124] [v2:127.0.0.1:6802/2331858124,v1:127.0.0.1:6803/2331858124] exists,up e1c6458e-4e37-4fc7-9862-0eeeb480d618 2026-02-25T22:21:09.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:124: TEST_ec_profile_blaum_roth_warning: for id in $(seq 0 2) 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:125: TEST_ec_profile_blaum_roth_warning: run_osd td/test-erasure-code-plugins 1 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/1 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:09.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:09.424 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:21:09.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:21:09.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/1 2026-02-25T22:21:09.426 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:21:09.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=31dafbe4-db96-48c5-971a-fefd3b1a6868 2026-02-25T22:21:09.427 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 31dafbe4-db96-48c5-971a-fefd3b1a6868 2026-02-25T22:21:09.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 31dafbe4-db96-48c5-971a-fefd3b1a6868' 2026-02-25T22:21:09.427 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:21:09.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDVdZ9pqvAxGhAAPUg8OTVeOKgWuWsQgxZjjA== 2026-02-25T22:21:09.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDVdZ9pqvAxGhAAPUg8OTVeOKgWuWsQgxZjjA=="}' 2026-02-25T22:21:09.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 31dafbe4-db96-48c5-971a-fefd3b1a6868 -i td/test-erasure-code-plugins/1/new.json 2026-02-25T22:21:09.652 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:21:09.661 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/1/new.json 2026-02-25T22:21:09.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVdZ9pqvAxGhAAPUg8OTVeOKgWuWsQgxZjjA== --osd-uuid 31dafbe4-db96-48c5-971a-fefd3b1a6868 2026-02-25T22:21:09.677 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:09.675+0000 7f788ea64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:09.678 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:09.675+0000 7f788ea64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:09.679 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:09.679+0000 7f788ea64a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:09.680 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:09.679+0000 7f788ea64a40 -1 bdev(0x560d9199a400 td/test-erasure-code-plugins/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:21:09.680 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:09.679+0000 7f788ea64a40 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2026-02-25T22:21:10.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2026-02-25T22:21:10.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:21:10.343 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:21:10.343 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:21:10.343 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:21:10.633 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:21:10.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:21:10.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:21:10.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:21:10.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:21:10.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:21:10.647 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:10.647+0000 7f3223415a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:10.649 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:10.647+0000 7f3223415a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:10.650 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:10.647+0000 7f3223415a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:10.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:21:10.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:10.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:21:10.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:10.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:10.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:10.844 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:10.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:10.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:10.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:21:11.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:11.064 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:11.063+0000 7f3223415a40 -1 Falling back to public interface 2026-02-25T22:21:11.709 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:11.707+0000 7f3223415a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:21:12.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:12.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:12.054 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:21:12.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:12.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:12.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:21:12.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:13.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:13.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:13.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:21:13.265 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:21:13.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:13.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:21:13.476 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1007600993,v1:127.0.0.1:6811/1007600993] [v2:127.0.0.1:6812/1007600993,v1:127.0.0.1:6813/1007600993] exists,up 31dafbe4-db96-48c5-971a-fefd3b1a6868 2026-02-25T22:21:13.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:13.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:13.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:13.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:124: TEST_ec_profile_blaum_roth_warning: for id in $(seq 0 2) 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:125: TEST_ec_profile_blaum_roth_warning: run_osd td/test-erasure-code-plugins 2 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/2 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2026-02-25T22:21:13.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:13.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:21:13.482 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:21:13.482 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:21:13.482 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:21:13.482 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:13.482 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:13.482 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:13.482 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:13.482 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:21:13.482 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:21:13.482 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/2 2026-02-25T22:21:13.483 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:21:13.483 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3d40e7de-25d5-434c-800b-8cc8c473c2bd 2026-02-25T22:21:13.483 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 3d40e7de-25d5-434c-800b-8cc8c473c2bd 2026-02-25T22:21:13.484 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 3d40e7de-25d5-434c-800b-8cc8c473c2bd' 2026-02-25T22:21:13.484 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:21:13.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDZdZ9pVJqQHRAA+jcPBYmWuRIPtmPP+uflTg== 2026-02-25T22:21:13.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDZdZ9pVJqQHRAA+jcPBYmWuRIPtmPP+uflTg=="}' 2026-02-25T22:21:13.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3d40e7de-25d5-434c-800b-8cc8c473c2bd -i td/test-erasure-code-plugins/2/new.json 2026-02-25T22:21:13.706 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:21:13.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/2/new.json 2026-02-25T22:21:13.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZdZ9pVJqQHRAA+jcPBYmWuRIPtmPP+uflTg== --osd-uuid 3d40e7de-25d5-434c-800b-8cc8c473c2bd 2026-02-25T22:21:13.741 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:13.739+0000 7f8c22694a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:13.742 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:13.739+0000 7f8c22694a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:13.744 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:13.743+0000 7f8c22694a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:13.744 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:13.743+0000 7f8c22694a40 -1 bdev(0x55cefb5b0400 td/test-erasure-code-plugins/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:21:13.744 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:13.743+0000 7f8c22694a40 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2026-02-25T22:21:14.443 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2026-02-25T22:21:14.443 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:21:14.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:21:14.444 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:21:14.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:21:14.727 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:21:14.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:21:14.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:21:14.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:21:14.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:21:14.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:21:14.743 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:14.743+0000 7f30c6b36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:14.745 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:14.743+0000 7f30c6b36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:14.746 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:14.743+0000 7f30c6b36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:14.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:21:14.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:14.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:21:14.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:14.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:14.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:14.935 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:14.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:14.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:14.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:21:15.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:15.181 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:15.179+0000 7f30c6b36a40 -1 Falling back to public interface 2026-02-25T22:21:15.901 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:15.899+0000 7f30c6b36a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:21:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:16.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:16.148 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:21:16.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:16.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:21:16.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:17.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:17.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:17.368 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:21:17.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:21:17.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:17.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:21:17.582 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1066447778,v1:127.0.0.1:6819/1066447778] [v2:127.0.0.1:6820/1066447778,v1:127.0.0.1:6821/1066447778] exists,up 3d40e7de-25d5-434c-800b-8cc8c473c2bd 2026-02-25T22:21:17.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:17.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:17.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:17.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:127: TEST_ec_profile_blaum_roth_warning: create_rbd_pool 2026-02-25T22:21:17.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:21:17.786 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' does not exist 2026-02-25T22:21:17.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:21:17.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:21:18.027 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:21:18.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:21:19.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:21:19.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:128: TEST_ec_profile_blaum_roth_warning: wait_for_clean 2026-02-25T22:21:19.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:21:19.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:21:19.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:21:19.321 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:21:19.321 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:21:19.321 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:21:19.321 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:21:19.321 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:21:19.321 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:21:19.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:21:19.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:21:19.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:21:19.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:21:19.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:21:19.421 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:21:19.632 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:21:19.633 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:21:19.633 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:21:19.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:21:19.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:21:19.633 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:21:19.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-25T22:21:19.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-25T22:21:19.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485' 2026-02-25T22:21:19.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:21:19.713 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:21:19.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-25T22:21:19.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-25T22:21:19.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485 1-38654705668' 2026-02-25T22:21:19.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:21:19.791 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:21:19.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574851 2026-02-25T22:21:19.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574851 2026-02-25T22:21:19.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485 1-38654705668 2-55834574851' 2026-02-25T22:21:19.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:21:19.867 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836485 2026-02-25T22:21:19.867 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:21:19.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:21:19.869 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-25T22:21:19.869 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:21:19.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836485 2026-02-25T22:21:19.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-25T22:21:19.871 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 21474836485 2026-02-25T22:21:19.871 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:21:20.080 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836482 -lt 21474836485 2026-02-25T22:21:20.080 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:21:21.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:21:21.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:21:21.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-25T22:21:21.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:21:21.295 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705668 2026-02-25T22:21:21.296 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:21:21.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:21:21.298 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-25T22:21:21.298 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:21:21.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705668 2026-02-25T22:21:21.299 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 38654705668 2026-02-25T22:21:21.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-25T22:21:21.300 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:21:21.509 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705668 -lt 38654705668 2026-02-25T22:21:21.509 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:21:21.510 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574851 2026-02-25T22:21:21.510 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:21:21.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:21:21.512 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574851 2026-02-25T22:21:21.512 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:21:21.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574851 2026-02-25T22:21:21.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574851' 2026-02-25T22:21:21.514 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 55834574851 2026-02-25T22:21:21.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:21:21.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574851 -lt 55834574851 2026-02-25T22:21:21.723 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:21:21.724 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:21:21.724 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:21:22.004 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:21:22.004 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:21:22.005 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:21:22.005 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:21:22.005 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:21:22.005 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:21:22.005 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:21:22.006 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:21:22.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:21:22.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:21:22.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:21:22.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:21:22.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:21:22.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:21:22.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:21:22.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:130: TEST_ec_profile_blaum_roth_warning: echo 'Starting test for blaum-roth profile health warning' 2026-02-25T22:21:22.485 INFO:tasks.workunit.client.0.trial125.stdout:Starting test for blaum-roth profile health warning 2026-02-25T22:21:22.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:133: TEST_ec_profile_blaum_roth_warning: ceph osd erasure-code-profile set prof- plugin=jerasure k=3 m=1 technique=blaum_roth w=7 --yes-i-really-mean-it --force 2026-02-25T22:21:22.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:135: TEST_ec_profile_blaum_roth_warning: ceph osd erasure-code-profile set test crush-device-class= crush-failure-domain=osd crush-num-failure-domains=0 crush-osds-per-failure-domain=0 crush-root=default k=4 l=3 m=2 name=lrcprofile plugin=lrc 2026-02-25T22:21:23.049 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:136: TEST_ec_profile_blaum_roth_warning: get_asok_path mon.a 2026-02-25T22:21:23.049 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:23.049 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:23.050 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:23.050 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:23.050 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:23.050 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:23.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:136: TEST_ec_profile_blaum_roth_warning: CEPH_ARGS= 2026-02-25T22:21:23.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:136: TEST_ec_profile_blaum_roth_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:21:23.098 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:21:23.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:137: TEST_ec_profile_blaum_roth_warning: sleep 10 2026-02-25T22:21:33.105 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:138: TEST_ec_profile_blaum_roth_warning: grep -F '1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:21:33.109 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:25.671+0000 7fe422a88640 0 log_channel(cluster) log [WRN] : Health check failed: 1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption (BLAUM_ROTH_W_IS_NOT_PRIME) 2026-02-25T22:21:33.109 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:25.671+0000 7fe420283640 10 mon.a@0(leader).log v82 logging 2026-02-25T22:21:25.673743+0000 mon.a (mon.0) 152 : cluster [WRN] Health check failed: 1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption (BLAUM_ROTH_W_IS_NOT_PRIME) 2026-02-25T22:21:33.109 INFO:tasks.workunit.client.0.trial125.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T22:21:33.110 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:25.719+0000 7fe41da7e640 7 mon.a@0(leader).log v83 update_from_paxos applying incremental log 83 2026-02-25T22:21:25.673743+0000 mon.a (mon.0) 152 : cluster [WRN] Health check failed: 1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption (BLAUM_ROTH_W_IS_NOT_PRIME) 2026-02-25T22:21:33.110 INFO:tasks.workunit.client.0.trial125.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T22:21:33.110 INFO:tasks.workunit.client.0.trial125.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T22:21:33.110 INFO:tasks.workunit.client.0.trial125.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T22:21:33.110 INFO:tasks.workunit.client.0.trial125.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T22:21:33.110 INFO:tasks.workunit.client.0.trial125.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T22:21:33.110 INFO:tasks.workunit.client.0.trial125.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T22:21:33.110 INFO:tasks.workunit.client.0.trial125.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T22:21:33.110 INFO:tasks.workunit.client.0.trial125.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T22:21:33.110 INFO:tasks.workunit.client.0.trial125.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T22:21:33.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:139: TEST_ec_profile_blaum_roth_warning: grep -F 'w+1=8 for the EC profile prof- is not prime and could lead to data corruption' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:21:33.112 INFO:tasks.workunit.client.0.trial125.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T22:21:33.112 INFO:tasks.workunit.client.0.trial125.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T22:21:33.112 INFO:tasks.workunit.client.0.trial125.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T22:21:33.112 INFO:tasks.workunit.client.0.trial125.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T22:21:33.112 INFO:tasks.workunit.client.0.trial125.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T22:21:33.113 INFO:tasks.workunit.client.0.trial125.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T22:21:33.113 INFO:tasks.workunit.client.0.trial125.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T22:21:33.113 INFO:tasks.workunit.client.0.trial125.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T22:21:33.113 INFO:tasks.workunit.client.0.trial125.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T22:21:33.113 INFO:tasks.workunit.client.0.trial125.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T22:21:33.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:141: TEST_ec_profile_blaum_roth_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:21:33.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:33.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:33.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:21:33.114 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:33.114 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:33.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:33.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:33.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:33.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:33.222 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:33.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:33.223 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:33.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:33.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:33.225 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:33.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:33.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:33.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:33.226 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:33.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:33.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:33.229 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:21:33.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:33.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:33.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:33.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:21:33.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:33.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:33.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:142: TEST_ec_profile_blaum_roth_warning: return 0 2026-02-25T22:21:33.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-25T22:21:33.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2026-02-25T22:21:33.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:33.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2026-02-25T22:21:33.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:33.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:21:33.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:33.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:33.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:21:33.243 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:33.243 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:33.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:33.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:33.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:33.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:33.244 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:33.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:33.245 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:33.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:33.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:33.246 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:33.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:33.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:33.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:33.248 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:33.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:33.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:33.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:21:33.251 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:33.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:33.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:33.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:21:33.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:33.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:33.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:21:33.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:21:33.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:33.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:33.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:21:33.256 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:21:33.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:21:33.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:21:33.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:21:33.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2026-02-25T22:21:33.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:33.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:21:33.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:21:33.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:21:33.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:21:33.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2026-02-25T22:21:33.278 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:21:33.278 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:33.278 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:33.278 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:33.278 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:33.278 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:33.279 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:33.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:21:33.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:21:33.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:21:33.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:33.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:33.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:21:33.311 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:21:33.311 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:33.311 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:33.311 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:33.311 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:33.311 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:33.311 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:33.311 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:33.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:33.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:21:33.366 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:21:33.366 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:33.367 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:33.367 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:21:33.368 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:21:33.368 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:33.368 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:33.368 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:33.368 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:33.368 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:33.368 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:33.368 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:33.369 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:33.369 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:21:33.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:21:33.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:33.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:21:33.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:21:33.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:21:33.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:21:33.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:21:33.537 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:21:33.537 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:33.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:33.538 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:33.538 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:33.538 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:33.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:33.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:33.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:33.556 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2026-02-25T22:21:33.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2026-02-25T22:21:33.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2026-02-25T22:21:33.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:33.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:21:33.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:21:33.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:21:33.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:33.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:33.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:33.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:33.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:21:33.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:21:33.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:21:33.563 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:21:33.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ef481ec7-6cbb-446b-8213-4dea1da18051 2026-02-25T22:21:33.563 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 ef481ec7-6cbb-446b-8213-4dea1da18051 2026-02-25T22:21:33.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ef481ec7-6cbb-446b-8213-4dea1da18051' 2026-02-25T22:21:33.564 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:21:33.571 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDtdZ9pCr49IhAAnzoPPeKfOLDdkEzbe8IXNA== 2026-02-25T22:21:33.571 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDtdZ9pCr49IhAAnzoPPeKfOLDdkEzbe8IXNA=="}' 2026-02-25T22:21:33.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ef481ec7-6cbb-446b-8213-4dea1da18051 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:21:33.678 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:33.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:21:33.687 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtdZ9pCr49IhAAnzoPPeKfOLDdkEzbe8IXNA== --osd-uuid ef481ec7-6cbb-446b-8213-4dea1da18051 2026-02-25T22:21:33.701 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:33.699+0000 7fd514637a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:33.702 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:33.699+0000 7fd514637a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:33.703 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:33.699+0000 7fd514637a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:33.703 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:33.703+0000 7fd514637a40 -1 bdev(0x55f13ed70400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:21:33.703 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:33.703+0000 7fd514637a40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:21:34.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:21:34.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:21:34.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:21:34.516 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:21:34.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:21:34.637 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:21:34.637 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:21:34.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:21:34.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:21:34.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:21:34.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:21:34.660 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:34.659+0000 7f0e40bb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:34.662 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:34.659+0000 7f0e40bb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:34.663 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:34.663+0000 7f0e40bb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:34.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:21:34.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:34.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:21:34.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:34.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:34.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:34.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:34.747 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:34.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:34.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:34.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:35.092 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:35.091+0000 7f0e40bb8a40 -1 Falling back to public interface 2026-02-25T22:21:35.807 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:35.807+0000 7f0e40bb8a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:21:35.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:35.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:35.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:35.853 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:21:35.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:35.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:36.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:37.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:37.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:37.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:21:37.065 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:21:37.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:37.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:37.273 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3154358043,v1:127.0.0.1:6801/3154358043] [v2:127.0.0.1:6802/3154358043,v1:127.0.0.1:6803/3154358043] exists,up ef481ec7-6cbb-446b-8213-4dea1da18051 2026-02-25T22:21:37.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:37.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:37.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:37.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2026-02-25T22:21:37.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2026-02-25T22:21:37.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:37.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:21:37.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:21:37.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/1 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:21:37.278 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:37.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:37.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:37.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:37.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:21:37.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:21:37.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/1 2026-02-25T22:21:37.281 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:21:37.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=122e103d-a23e-4e11-a6d1-7c3a8bfef0ee 2026-02-25T22:21:37.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 122e103d-a23e-4e11-a6d1-7c3a8bfef0ee' 2026-02-25T22:21:37.281 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 122e103d-a23e-4e11-a6d1-7c3a8bfef0ee 2026-02-25T22:21:37.281 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:21:37.290 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDxdZ9pleZyERAAUfJ8M3gEN1cJpR7T1ChZcQ== 2026-02-25T22:21:37.290 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDxdZ9pleZyERAAUfJ8M3gEN1cJpR7T1ChZcQ=="}' 2026-02-25T22:21:37.290 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 122e103d-a23e-4e11-a6d1-7c3a8bfef0ee -i td/test-erasure-code-plugins/1/new.json 2026-02-25T22:21:37.504 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:21:37.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/1/new.json 2026-02-25T22:21:37.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxdZ9pleZyERAAUfJ8M3gEN1cJpR7T1ChZcQ== --osd-uuid 122e103d-a23e-4e11-a6d1-7c3a8bfef0ee 2026-02-25T22:21:37.529 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:37.527+0000 7f96fef26a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:37.530 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:37.527+0000 7f96fef26a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:37.531 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:37.531+0000 7f96fef26a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:37.532 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:37.531+0000 7f96fef26a40 -1 bdev(0x555ee6c7e400 td/test-erasure-code-plugins/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:21:37.532 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:37.531+0000 7f96fef26a40 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2026-02-25T22:21:38.205 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2026-02-25T22:21:38.205 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:21:38.206 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:21:38.206 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:21:38.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:21:38.492 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:21:38.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:21:38.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:21:38.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:21:38.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:21:38.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:21:38.507 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:38.507+0000 7f6fbdd0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:38.509 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:38.507+0000 7f6fbdd0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:38.510 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:38.507+0000 7f6fbdd0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:38.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:21:38.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:38.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:21:38.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:38.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:38.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:38.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:38.699 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:38.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:38.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:21:38.865 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:38.863+0000 7f6fbdd0da40 -1 Falling back to public interface 2026-02-25T22:21:38.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:39.602 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:39.599+0000 7f6fbdd0da40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:21:39.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:39.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:39.915 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:21:39.915 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:39.915 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:39.915 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:21:40.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:41.130 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:41.130 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:41.130 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:21:41.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:21:41.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:41.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:21:41.335 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/799529429,v1:127.0.0.1:6811/799529429] [v2:127.0.0.1:6812/799529429,v1:127.0.0.1:6813/799529429] exists,up 122e103d-a23e-4e11-a6d1-7c3a8bfef0ee 2026-02-25T22:21:41.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:41.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:41.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/2 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:41.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:41.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:21:41.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/2 2026-02-25T22:21:41.342 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:21:41.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=acfe8b41-1ad8-4db7-b98a-a10ea325d91d 2026-02-25T22:21:41.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 acfe8b41-1ad8-4db7-b98a-a10ea325d91d' 2026-02-25T22:21:41.343 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 acfe8b41-1ad8-4db7-b98a-a10ea325d91d 2026-02-25T22:21:41.343 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:21:41.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD1dZ9pCtIkFRAATXe9FcNQgrZZR0x4J7spqw== 2026-02-25T22:21:41.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD1dZ9pCtIkFRAATXe9FcNQgrZZR0x4J7spqw=="}' 2026-02-25T22:21:41.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new acfe8b41-1ad8-4db7-b98a-a10ea325d91d -i td/test-erasure-code-plugins/2/new.json 2026-02-25T22:21:41.560 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:21:41.569 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/2/new.json 2026-02-25T22:21:41.570 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1dZ9pCtIkFRAATXe9FcNQgrZZR0x4J7spqw== --osd-uuid acfe8b41-1ad8-4db7-b98a-a10ea325d91d 2026-02-25T22:21:41.586 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:41.583+0000 7fc3637d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:41.588 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:41.587+0000 7fc3637d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:41.589 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:41.587+0000 7fc3637d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:41.590 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:41.587+0000 7fc3637d0a40 -1 bdev(0x555812b42400 td/test-erasure-code-plugins/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:21:41.590 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:41.587+0000 7fc3637d0a40 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2026-02-25T22:21:42.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2026-02-25T22:21:42.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:21:42.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:21:42.276 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:21:42.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:21:42.565 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:21:42.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:21:42.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:21:42.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:21:42.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:21:42.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:21:42.588 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:42.587+0000 7fdf8b512a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:42.590 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:42.587+0000 7fdf8b512a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:42.591 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:42.591+0000 7fdf8b512a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:42.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:21:42.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:42.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:21:42.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:42.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:42.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:42.779 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:42.780 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:42.780 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:42.780 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:21:42.929 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:42.927+0000 7fdf8b512a40 -1 Falling back to public interface 2026-02-25T22:21:42.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:43.528 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:43.527+0000 7fdf8b512a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:21:43.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:43.991 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:43.991 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:21:43.991 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:43.992 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:43.992 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:21:44.194 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2733032524,v1:127.0.0.1:6819/2733032524] [v2:127.0.0.1:6820/2733032524,v1:127.0.0.1:6821/2733032524] exists,up acfe8b41-1ad8-4db7-b98a-a10ea325d91d 2026-02-25T22:21:44.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:44.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:44.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:44.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2026-02-25T22:21:44.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:21:44.402 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' does not exist 2026-02-25T22:21:44.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:21:44.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:21:44.657 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:21:44.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:21:45.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:21:45.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2026-02-25T22:21:45.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:21:45.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:21:45.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:21:45.950 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:21:45.950 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:21:45.950 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:21:45.950 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:21:45.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:21:45.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:21:46.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:21:46.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:21:46.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:21:46.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:21:46.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:21:46.053 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:21:46.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:21:46.262 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:21:46.262 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:21:46.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:21:46.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:21:46.263 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:21:46.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-25T22:21:46.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-25T22:21:46.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485' 2026-02-25T22:21:46.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:21:46.339 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:21:46.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-25T22:21:46.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-25T22:21:46.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485 1-38654705669' 2026-02-25T22:21:46.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:21:46.417 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:21:46.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574851 2026-02-25T22:21:46.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574851 2026-02-25T22:21:46.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485 1-38654705669 2-55834574851' 2026-02-25T22:21:46.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:21:46.495 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836485 2026-02-25T22:21:46.495 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:21:46.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:21:46.496 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-25T22:21:46.496 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:21:46.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836485 2026-02-25T22:21:46.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-25T22:21:46.498 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 21474836485 2026-02-25T22:21:46.499 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:21:46.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-25T22:21:46.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:21:47.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:21:47.717 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:21:47.929 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-25T22:21:47.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:21:47.930 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705669 2026-02-25T22:21:47.930 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:21:47.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:21:47.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-25T22:21:47.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:21:47.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705669 2026-02-25T22:21:47.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-25T22:21:47.934 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 38654705669 2026-02-25T22:21:47.934 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:21:48.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-25T22:21:48.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:21:48.150 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574851 2026-02-25T22:21:48.150 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:21:48.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:21:48.152 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574851 2026-02-25T22:21:48.152 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:21:48.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574851 2026-02-25T22:21:48.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574851' 2026-02-25T22:21:48.154 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 55834574851 2026-02-25T22:21:48.154 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:21:48.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574851 -lt 55834574851 2026-02-25T22:21:48.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:21:48.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:21:48.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:21:48.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:21:48.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:21:48.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:21:48.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:21:48.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:21:48.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:21:48.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:21:48.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:21:48.858 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:21:48.858 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:21:48.859 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:21:48.859 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:21:49.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:21:49.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:21:49.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:21:49.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2026-02-25T22:21:49.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2026-02-25T22:21:49.415 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T22:21:49.415 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:49.416 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:49.416 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:49.417 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:49.417 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:49.417 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:49.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T22:21:49.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:21:49.465 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:21:49.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:21:49.473 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:49.303+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e20 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2026-02-25T22:21:49.473 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:49.407+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e21 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2026-02-25T22:21:49.473 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:49.407+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e21 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2026-02-25T22:21:49.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2026-02-25T22:21:49.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2026-02-25T22:21:49.776 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T22:21:49.776 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:49.776 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:49.777 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:49.777 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:49.778 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:49.778 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:49.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T22:21:49.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:21:49.825 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:21:49.831 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:21:49.833 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:49.623+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e21 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2026-02-25T22:21:49.834 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:49.767+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e22 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2026-02-25T22:21:49.834 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:49.767+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e22 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2026-02-25T22:21:49.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2026-02-25T22:21:49.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2026-02-25T22:21:50.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T22:21:50.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:50.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:50.099 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:50.099 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:50.100 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:50.100 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:50.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T22:21:50.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:21:50.151 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:21:50.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:21:50.160 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:49.983+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e22 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2026-02-25T22:21:50.160 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:50.091+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2026-02-25T22:21:50.160 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:50.091+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2026-02-25T22:21:50.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2026-02-25T22:21:50.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2026-02-25T22:21:50.434 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T22:21:50.434 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:50.434 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:50.434 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:50.434 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:50.434 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:50.435 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:50.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T22:21:50.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:21:50.485 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:21:50.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:21:50.493 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:50.315+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2026-02-25T22:21:50.493 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:50.427+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2026-02-25T22:21:50.493 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:50.427+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2026-02-25T22:21:50.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2026-02-25T22:21:50.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2026-02-25T22:21:50.737 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T22:21:50.737 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:50.737 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:50.738 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:50.738 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:50.738 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:50.738 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:50.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T22:21:50.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:21:50.786 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:21:50.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:21:50.794 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:50.647+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2026-02-25T22:21:50.794 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:50.727+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2026-02-25T22:21:50.794 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:50.727+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2026-02-25T22:21:50.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2026-02-25T22:21:50.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2026-02-25T22:21:51.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T22:21:51.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:51.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:51.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:51.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:51.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:51.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T22:21:51.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:21:51.113 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:21:51.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:21:51.121 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:50.947+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2026-02-25T22:21:51.121 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:51.055+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2026-02-25T22:21:51.121 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:21:51.055+0000 7f5bf0ae7640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2026-02-25T22:21:51.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:21:51.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:51.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:51.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:21:51.123 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:51.123 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:51.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:51.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:51.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:51.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:51.231 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:51.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:51.233 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:51.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:51.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:51.234 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:51.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:51.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:51.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:51.236 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:51.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:51.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:51.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:21:51.248 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:51.248 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.248 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:51.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:21:51.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:51.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:51.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-25T22:21:51.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2026-02-25T22:21:51.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:51.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-25T22:21:51.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-25T22:21:51.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:51.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:21:51.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:51.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:51.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:21:51.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:51.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:51.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:51.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:51.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:51.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:51.253 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:51.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:51.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:51.255 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:51.255 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:51.256 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:51.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:51.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:51.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:51.257 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:51.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:51.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:51.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:21:51.261 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:51.261 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.261 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:51.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:21:51.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:51.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:51.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:21:51.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:21:51.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:51.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:21:51.266 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:21:51.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:21:51.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:21:51.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:21:51.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2026-02-25T22:21:51.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:51.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:21:51.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:21:51.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:21:51.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:21:51.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2026-02-25T22:21:51.287 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:21:51.287 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:51.287 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:51.287 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:51.287 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.288 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:51.288 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:51.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2026-02-25T22:21:51.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:21:51.315 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:21:51.316 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:51.316 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:51.316 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:21:51.317 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:21:51.317 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:51.317 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:51.317 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:51.317 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:51.317 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.317 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:51.317 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:51.318 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:51.318 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:21:51.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:21:51.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:51.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:51.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:21:51.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:21:51.374 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:51.374 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:51.375 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:51.375 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:51.375 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.375 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:51.375 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:51.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:51.376 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:21:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:21:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:21:51.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:21:51.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:21:51.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:21:51.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:21:51.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:21:51.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:51.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:51.538 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:51.538 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.538 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:51.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:51.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:51.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:51.559 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-25T22:21:51.559 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:51.559 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:51.559 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:51.559 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.559 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:51.560 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:51.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T22:21:51.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:21:51.616 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:21:51.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2026-02-25T22:21:51.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:51.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:21:51.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:21:51.626 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:21:51.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:51.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:51.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:51.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:51.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:21:51.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2026-02-25T22:21:51.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:21:51.629 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:21:51.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9af2855e-b9fb-4127-b7b5-672f44d8e408 2026-02-25T22:21:51.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9af2855e-b9fb-4127-b7b5-672f44d8e408' 2026-02-25T22:21:51.630 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 9af2855e-b9fb-4127-b7b5-672f44d8e408 2026-02-25T22:21:51.630 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:21:51.637 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD/dZ9p+8spJhAArGTHl9SLyikqkw9oTWMHnA== 2026-02-25T22:21:51.637 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD/dZ9p+8spJhAArGTHl9SLyikqkw9oTWMHnA=="}' 2026-02-25T22:21:51.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9af2855e-b9fb-4127-b7b5-672f44d8e408 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:21:51.746 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:51.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:21:51.755 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQD/dZ9p+8spJhAArGTHl9SLyikqkw9oTWMHnA== --osd-uuid 9af2855e-b9fb-4127-b7b5-672f44d8e408 2026-02-25T22:21:51.771 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:51.775+0000 7fe82d7caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:51.773 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:51.775+0000 7fe82d7caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:51.774 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:51.775+0000 7fe82d7caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:51.774 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:51.775+0000 7fe82d7caa40 -1 bdev(0x55898bfc8400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:21:51.775 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:51.775+0000 7fe82d7caa40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:21:52.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:21:52.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:21:52.537 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:21:52.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:21:52.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:21:52.654 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:21:52.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:21:52.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2026-02-25T22:21:52.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:21:52.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:21:52.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:21:52.678 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:52.679+0000 7f197fa7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:52.680 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:52.683+0000 7f197fa7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:52.681 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:52.683+0000 7f197fa7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:52.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:21:52.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:52.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:21:52.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:52.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:52.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:52.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:52.764 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:52.765 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:52.765 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:52.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:53.117 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:53.119+0000 7f197fa7aa40 -1 Falling back to public interface 2026-02-25T22:21:53.844 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:53.847+0000 7f197fa7aa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:21:53.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:53.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:53.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:53.874 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:21:53.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:53.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:54.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:55.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:55.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:55.083 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:21:55.084 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:21:55.084 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:55.084 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:55.286 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1537304055,v1:127.0.0.1:6801/1537304055] [v2:127.0.0.1:6802/1537304055,v1:127.0.0.1:6803/1537304055] exists,up 9af2855e-b9fb-4127-b7b5-672f44d8e408 2026-02-25T22:21:55.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:55.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:55.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:55.288 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-25T22:21:55.288 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:55.288 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:55.288 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:55.288 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:55.288 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:55.289 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-osd.0.asok 2026-02-25T22:21:55.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T22:21:55.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-osd.0.asok log flush 2026-02-25T22:21:55.350 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:21:55.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:21:55.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-25T22:21:55.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:21:55.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:55.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:55.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:21:55.360 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:55.360 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:55.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:55.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:55.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:55.669 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:55.670 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:55.671 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:55.671 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:55.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:55.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:55.673 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:55.675 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:55.675 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:55.676 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:55.676 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:55.677 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:55.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:55.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:21:55.682 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:55.682 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:55.682 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:55.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:21:55.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:55.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:55.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-25T22:21:55.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-25T22:21:55.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:55.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:21:55.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:55.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:55.686 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:21:55.686 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:55.686 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:55.686 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:55.686 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:55.686 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:55.687 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:55.687 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:55.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:55.688 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:55.689 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:55.689 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:55.690 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:55.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:55.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:55.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:55.691 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:55.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:55.694 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:55.694 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:21:55.695 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:55.695 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:55.695 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:55.695 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:21:55.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:55.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:55.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:21:55.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:21:55.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:55.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:55.698 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:21:55.699 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:21:55.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:21:55.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:21:55.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:21:55.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2026-02-25T22:21:55.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:55.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:21:55.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:21:55.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:21:55.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:21:55.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2026-02-25T22:21:55.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:21:55.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:55.723 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:55.723 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:55.723 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:55.723 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:55.723 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:55.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2026-02-25T22:21:55.750 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:21:55.751 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:21:55.751 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:55.751 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:55.752 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:21:55.752 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:21:55.753 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:55.753 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:55.753 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:55.753 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:55.753 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:55.753 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:55.753 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:55.754 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:55.754 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:21:55.807 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:21:55.807 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:55.807 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:55.807 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:21:55.808 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:21:55.809 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:55.809 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:55.809 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:55.809 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:55.809 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:55.809 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:55.809 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:55.809 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:55.809 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:21:55.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:21:55.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:55.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:21:55.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:21:55.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:21:55.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:21:55.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:21:55.969 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:21:55.970 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:55.970 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:55.970 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:55.970 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:55.970 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:55.971 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:55.971 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:55.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:55.991 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-25T22:21:55.991 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:55.992 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:55.992 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:55.992 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:55.992 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:55.992 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:21:55.992 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T22:21:55.992 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:21:56.055 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:21:56.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2026-02-25T22:21:56.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:56.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:21:56.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:21:56.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:21:56.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:21:56.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:21:56.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:56.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:56.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:56.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:21:56.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:21:56.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:56.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:21:56.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:21:56.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:21:56.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:21:56.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:56.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:56.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:56.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:56.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:21:56.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2026-02-25T22:21:56.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:21:56.068 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:21:56.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b9f3bbff-e7a0-4a09-87a6-49f690ac00ec 2026-02-25T22:21:56.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b9f3bbff-e7a0-4a09-87a6-49f690ac00ec' 2026-02-25T22:21:56.068 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 b9f3bbff-e7a0-4a09-87a6-49f690ac00ec 2026-02-25T22:21:56.069 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:21:56.077 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAEdp9pJwfIBBAAmeylw2xmjAfSxzdIKFJJjg== 2026-02-25T22:21:56.077 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAEdp9pJwfIBBAAmeylw2xmjAfSxzdIKFJJjg=="}' 2026-02-25T22:21:56.077 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b9f3bbff-e7a0-4a09-87a6-49f690ac00ec -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:21:56.187 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:56.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:21:56.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQAEdp9pJwfIBBAAmeylw2xmjAfSxzdIKFJJjg== --osd-uuid b9f3bbff-e7a0-4a09-87a6-49f690ac00ec 2026-02-25T22:21:56.211 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:56.215+0000 7fc6aa46aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:56.213 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:56.215+0000 7fc6aa46aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:56.214 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:56.215+0000 7fc6aa46aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:56.214 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:56.215+0000 7fc6aa46aa40 -1 bdev(0x56276e2cc400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:21:56.215 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:56.215+0000 7fc6aa46aa40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:21:57.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:21:57.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:21:57.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:21:57.008 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:21:57.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:21:57.124 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:21:57.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:21:57.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2026-02-25T22:21:57.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:21:57.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:21:57.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:21:57.146 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:57.147+0000 7fef1693da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:57.148 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:57.151+0000 7fef1693da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:57.150 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:57.151+0000 7fef1693da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:57.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:21:57.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:57.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:21:57.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:57.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:57.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:57.232 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:21:57.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:57.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:57.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:57.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:57.629 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:57.631+0000 7fef1693da40 -1 Falling back to public interface 2026-02-25T22:21:58.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:58.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:58.339 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:21:58.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:58.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:58.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:58.531 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:21:58.530+0000 7fef1693da40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:21:58.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:59.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:59.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:59.546 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:21:59.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:21:59.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:59.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:59.753 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3382638311,v1:127.0.0.1:6801/3382638311] [v2:127.0.0.1:6802/3382638311,v1:127.0.0.1:6803/3382638311] exists,up b9f3bbff-e7a0-4a09-87a6-49f690ac00ec 2026-02-25T22:21:59.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:59.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:59.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:59.754 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-25T22:21:59.754 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:59.754 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:59.754 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:59.754 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:59.754 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:59.755 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-osd.0.asok 2026-02-25T22:21:59.755 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T22:21:59.755 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-osd.0.asok log flush 2026-02-25T22:21:59.807 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:21:59.813 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:21:59.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-25T22:21:59.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:21:59.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:59.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:59.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:21:59.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:59.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:59.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:59.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:59.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:59.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:59.926 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:59.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:59.927 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:59.928 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:59.928 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:59.929 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:59.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:59.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:59.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:59.930 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:59.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:59.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:59.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:21:59.936 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:59.936 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:59.936 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:59.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:21:59.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:59.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:59.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:59.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:59.940 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:59.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:59.942 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:59.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:59.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:59.944 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:59.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:59.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:59.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:59.945 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:59.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:59.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:59.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:21:59.949 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:59.949 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:59.949 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:59.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:21:59.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:59.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:59.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:21:59.952 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:21:59.952 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:59.952 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:59.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:21:59.954 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:21:59.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:21:59.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:21:59.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:21:59.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2026-02-25T22:21:59.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:21:59.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:21:59.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:21:59.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:21:59.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:21:59.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2026-02-25T22:21:59.975 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:21:59.975 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:59.975 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:59.976 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:59.976 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:59.976 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:21:59.976 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:21:59.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2026-02-25T22:22:00.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:00.005 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:00.005 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:00.005 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:00.005 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:00.006 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:00.006 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:00.006 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:00.006 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:00.007 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:00.007 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:00.007 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:00.007 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:00.007 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:00.007 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:22:00.061 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:00.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:00.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:00.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:00.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:00.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:00.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:00.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:00.063 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:00.064 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:00.064 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:00.064 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:00.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:00.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:00.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:22:00.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:00.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:00.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:00.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:00.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:22:00.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:00.226 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:00.226 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:00.226 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:00.227 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:00.227 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:00.227 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:00.227 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:00.228 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:00.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:00.246 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-25T22:22:00.246 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:00.246 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:00.246 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:00.247 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:00.247 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:00.247 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:00.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T22:22:00.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:22:00.298 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:00.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2026-02-25T22:22:00.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:00.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:00.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:00.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:00.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:22:00.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:00.308 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:00.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2026-02-25T22:22:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:22:00.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=282fb5fc-bc08-4a65-84be-67bb559de6cc 2026-02-25T22:22:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 282fb5fc-bc08-4a65-84be-67bb559de6cc' 2026-02-25T22:22:00.311 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 282fb5fc-bc08-4a65-84be-67bb559de6cc 2026-02-25T22:22:00.311 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:00.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAIdp9pHAAqExAASp7O0oP54/7XJf/8lMFnfA== 2026-02-25T22:22:00.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAIdp9pHAAqExAASp7O0oP54/7XJf/8lMFnfA=="}' 2026-02-25T22:22:00.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 282fb5fc-bc08-4a65-84be-67bb559de6cc -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:00.426 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:00.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:00.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQAIdp9pHAAqExAASp7O0oP54/7XJf/8lMFnfA== --osd-uuid 282fb5fc-bc08-4a65-84be-67bb559de6cc 2026-02-25T22:22:00.449 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:00.450+0000 7ff9d14c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:00.451 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:00.454+0000 7ff9d14c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:00.452 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:00.454+0000 7ff9d14c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:00.453 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:00.454+0000 7ff9d14c8a40 -1 bdev(0x5642c07c2400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:00.453 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:00.454+0000 7ff9d14c8a40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:22:01.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:22:01.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:01.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:01.236 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:22:01.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:01.355 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:22:01.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:01.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2026-02-25T22:22:01.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:01.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:01.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:01.377 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:01.378+0000 7fb14779ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:01.379 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:01.382+0000 7fb14779ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:01.380 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:01.382+0000 7fb14779ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:01.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:01.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:01.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:01.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:01.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:01.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:01.462 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:01.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:01.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:01.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:01.571 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:01.796 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:01.798+0000 7fb14779ea40 -1 Falling back to public interface 2026-02-25T22:22:02.488 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:02.490+0000 7fb14779ea40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:02.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:02.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:02.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:02.574 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:02.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:02.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:02.780 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:03.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:03.782 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:03.782 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:03.783 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:03.783 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:03.783 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:03.985 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/207466825,v1:127.0.0.1:6801/207466825] [v2:127.0.0.1:6802/207466825,v1:127.0.0.1:6803/207466825] exists,up 282fb5fc-bc08-4a65-84be-67bb559de6cc 2026-02-25T22:22:03.986 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:03.986 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:03.986 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:03.987 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-25T22:22:03.987 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:03.987 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:03.987 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:03.987 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:03.987 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:03.987 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-osd.0.asok 2026-02-25T22:22:03.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T22:22:03.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-osd.0.asok log flush 2026-02-25T22:22:04.040 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:04.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:22:04.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-25T22:22:04.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:22:04.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:04.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:04.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:04.050 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:04.050 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:04.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:04.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:04.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:04.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:04.158 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:04.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:04.159 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:04.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:04.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:04.161 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:04.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:04.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:04.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:04.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:04.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:04.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:04.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:04.169 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:04.169 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:04.169 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:04.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:04.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:04.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:04.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-25T22:22:04.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-25T22:22:04.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:04.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:22:04.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:04.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:04.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:04.172 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:04.172 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:04.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:04.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:04.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:04.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:04.174 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:04.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:04.175 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:04.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:04.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:04.177 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:04.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:04.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:04.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:04.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:04.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:04.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:04.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:04.182 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:04.182 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:04.182 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:04.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:04.183 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:04.183 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:04.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:22:04.185 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:04.185 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:04.185 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:04.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:22:04.187 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:04.188 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:22:04.188 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:04.188 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:22:04.188 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2026-02-25T22:22:04.188 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:04.188 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:22:04.188 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:22:04.188 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:22:04.188 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:22:04.189 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2026-02-25T22:22:04.209 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:22:04.209 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:04.209 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:04.210 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:04.210 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:04.210 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:04.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:04.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2026-02-25T22:22:04.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:04.237 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:04.237 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:04.237 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:04.237 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:04.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:04.238 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:04.239 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:04.239 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:04.239 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:04.239 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:04.239 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:04.239 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:04.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:04.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:22:04.293 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:04.293 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:04.293 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:04.293 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:04.294 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:04.295 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:04.295 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:04.295 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:04.295 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:04.295 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:04.295 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:04.295 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:04.295 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:04.296 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:04.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:22:04.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:04.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:04.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:04.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:04.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:22:04.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:04.457 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:04.457 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:04.457 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:04.458 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:04.458 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:04.458 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:04.458 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:04.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:04.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:04.478 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-25T22:22:04.478 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:04.479 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:04.479 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:04.479 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:04.479 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:04.479 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:04.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T22:22:04.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:22:04.528 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:04.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2026-02-25T22:22:04.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:04.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:04.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:04.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:04.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:22:04.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:22:04.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:04.537 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:04.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2026-02-25T22:22:04.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:22:04.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:04.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4abee708-5b42-4b2e-a3f5-bb984a94ecd3 2026-02-25T22:22:04.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4abee708-5b42-4b2e-a3f5-bb984a94ecd3' 2026-02-25T22:22:04.540 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 4abee708-5b42-4b2e-a3f5-bb984a94ecd3 2026-02-25T22:22:04.540 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:04.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAMdp9p4GXfIBAAHMXLaG+ZHOjYAD4XPk+KSQ== 2026-02-25T22:22:04.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAMdp9p4GXfIBAAHMXLaG+ZHOjYAD4XPk+KSQ=="}' 2026-02-25T22:22:04.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4abee708-5b42-4b2e-a3f5-bb984a94ecd3 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:04.658 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:04.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:04.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQAMdp9p4GXfIBAAHMXLaG+ZHOjYAD4XPk+KSQ== --osd-uuid 4abee708-5b42-4b2e-a3f5-bb984a94ecd3 2026-02-25T22:22:04.683 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:04.686+0000 7f6e74207a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:04.685 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:04.686+0000 7f6e74207a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:04.686 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:04.686+0000 7f6e74207a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:04.686 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:04.686+0000 7f6e74207a40 -1 bdev(0x5642b13d4400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:04.686 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:04.686+0000 7f6e74207a40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:22:05.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:22:05.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:05.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:05.465 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:22:05.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:05.583 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:22:05.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:05.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2026-02-25T22:22:05.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:05.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:05.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:05.607 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:05.606+0000 7faf52331a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:05.609 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:05.610+0000 7faf52331a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:05.610 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:05.610+0000 7faf52331a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:05.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:05.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:05.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:05.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:05.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:05.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:05.692 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:05.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:05.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:05.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:05.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:06.008 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:06.010+0000 7faf52331a40 -1 Falling back to public interface 2026-02-25T22:22:06.786 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:06.786+0000 7faf52331a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:06.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:06.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:06.797 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:06.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:06.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:06.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:07.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:08.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:08.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:08.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:08.007 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:08.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:08.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:08.217 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/879207174,v1:127.0.0.1:6801/879207174] [v2:127.0.0.1:6802/879207174,v1:127.0.0.1:6803/879207174] exists,up 4abee708-5b42-4b2e-a3f5-bb984a94ecd3 2026-02-25T22:22:08.218 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:08.218 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:08.218 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:08.218 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-25T22:22:08.218 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:08.218 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:08.219 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:08.219 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.219 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:08.219 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-osd.0.asok 2026-02-25T22:22:08.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T22:22:08.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-osd.0.asok log flush 2026-02-25T22:22:08.275 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:08.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:22:08.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-25T22:22:08.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:22:08.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:08.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:08.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:08.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:08.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:08.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:08.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:08.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:08.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:08.393 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:08.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:08.394 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:08.395 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:08.395 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:08.395 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:08.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:08.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:08.397 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:08.399 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:08.399 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:08.403 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:08.404 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.404 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:08.404 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:08.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:08.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:08.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2026-02-25T22:22:08.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-25T22:22:08.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2026-02-25T22:22:08.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:08.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2026-02-25T22:22:08.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:08.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:22:08.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:08.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:08.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:08.407 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:08.407 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:08.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:08.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:08.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:08.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:08.410 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:08.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:08.411 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:08.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:08.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:08.411 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:08.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:08.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:08.412 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.413 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:08.414 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:08.414 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:08.416 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:08.416 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.416 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:08.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:08.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:08.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:08.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:22:08.418 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:08.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:08.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:22:08.420 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:08.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:22:08.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:08.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:22:08.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2026-02-25T22:22:08.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:08.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:22:08.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:22:08.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:22:08.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:22:08.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2026-02-25T22:22:08.442 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:22:08.442 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:08.442 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:08.443 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:08.443 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.443 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:08.443 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:08.443 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:22:08.471 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:08.472 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:08.473 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:08.473 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:08.473 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:08.473 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:08.474 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:08.474 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:08.474 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:08.475 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:08.475 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.475 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:08.475 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:08.475 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:08.475 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:22:08.537 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:08.537 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:08.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:08.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.540 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:08.540 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:08.540 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:08.540 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:08.595 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2026-02-25T22:22:08.595 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:08.595 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:08.596 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:08.596 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.597 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:08.597 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:08.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2026-02-25T22:22:08.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:22:08.651 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:08.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:22:08.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:08.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:08.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:08.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:08.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:22:08.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:08.776 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:08.776 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:08.776 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:08.777 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:08.777 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.777 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:08.777 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:08.778 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:08.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:08.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2026-02-25T22:22:08.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:08.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:08.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:22:08.799 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:22:08.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:08.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:08.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:08.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:08.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:08.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:22:08.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:22:08.802 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:08.802 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a0a089e3-e73f-4d9b-9547-15ea57e26772 2026-02-25T22:22:08.802 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a0a089e3-e73f-4d9b-9547-15ea57e26772' 2026-02-25T22:22:08.802 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 a0a089e3-e73f-4d9b-9547-15ea57e26772 2026-02-25T22:22:08.802 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:08.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAQdp9p61JqMBAAF2Z3DKX9y8Ap728H0A31JA== 2026-02-25T22:22:08.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAQdp9p61JqMBAAF2Z3DKX9y8Ap728H0A31JA=="}' 2026-02-25T22:22:08.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a0a089e3-e73f-4d9b-9547-15ea57e26772 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:08.919 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:08.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:08.928 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQdp9p61JqMBAAF2Z3DKX9y8Ap728H0A31JA== --osd-uuid a0a089e3-e73f-4d9b-9547-15ea57e26772 2026-02-25T22:22:08.943 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:08.946+0000 7f0055d59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:08.945 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:08.946+0000 7f0055d59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:08.946 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:08.946+0000 7f0055d59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:08.946 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:08.946+0000 7f0055d59a40 -1 bdev(0x55afcc648400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:08.946 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:08.946+0000 7f0055d59a40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:22:09.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:22:09.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:09.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:09.683 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:22:09.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:09.799 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:22:09.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:09.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:22:09.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:09.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:09.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:09.823 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:09.822+0000 7f80b9aa2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:09.825 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:09.826+0000 7f80b9aa2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:09.826 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:09.826+0000 7f80b9aa2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:09.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:09.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:09.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:09.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:09.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:09.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:09.905 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:09.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:09.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:09.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:10.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:10.381 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:10.382+0000 7f80b9aa2a40 -1 Falling back to public interface 2026-02-25T22:22:11.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:11.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:11.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:11.010 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:11.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:11.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:11.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:11.365 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:11.366+0000 7f80b9aa2a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:12.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:12.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:12.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:12.116 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:12.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:12.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:12.319 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3948723087,v1:127.0.0.1:6801/3948723087] [v2:127.0.0.1:6802/3948723087,v1:127.0.0.1:6803/3948723087] exists,up a0a089e3-e73f-4d9b-9547-15ea57e26772 2026-02-25T22:22:12.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:12.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:12.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:12.321 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2026-02-25T22:22:12.321 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:12.321 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:12.321 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:12.321 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:12.321 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:12.321 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-osd.0.asok 2026-02-25T22:22:12.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2026-02-25T22:22:12.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-osd.0.asok log flush 2026-02-25T22:22:12.383 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:12.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:22:12.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2026-02-25T22:22:12.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2026-02-25T22:22:12.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:12.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:12.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:12.393 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:12.393 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:12.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:12.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:12.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:12.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:12.703 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:12.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:12.704 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:12.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:12.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:12.707 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:12.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:12.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:12.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:12.709 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:12.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:12.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:12.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:12.716 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:12.716 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:12.716 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:12.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2026-02-25T22:22:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-25T22:22:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2026-02-25T22:22:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T22:22:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T22:22:12.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:12.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:22:12.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:12.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:12.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:12.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:12.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:12.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:12.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:12.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:12.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:12.721 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:12.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:12.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:12.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:12.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:12.724 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:12.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:12.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:12.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:12.726 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:12.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:12.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:12.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:12.729 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:12.729 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:12.729 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:12.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:12.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:12.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:12.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:22:12.731 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:12.732 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:12.732 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:12.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:22:12.733 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:12.734 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:22:12.734 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:12.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:22:12.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2026-02-25T22:22:12.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:12.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:22:12.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:22:12.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:22:12.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:22:12.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2026-02-25T22:22:12.754 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:22:12.754 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:12.755 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:12.755 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:12.755 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:12.756 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:12.756 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:12.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2026-02-25T22:22:12.782 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:12.782 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:12.783 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:12.783 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:12.783 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:12.784 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:12.784 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:12.784 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:12.784 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:12.784 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:12.784 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:12.785 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:12.785 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:12.785 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:12.785 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:22:12.839 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:12.839 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:12.839 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:12.839 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:12.840 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:12.841 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:12.841 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:12.841 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:12.841 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:12.841 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:12.841 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:12.841 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:12.841 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:12.842 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:12.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:22:12.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:12.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:12.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:12.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:12.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:22:12.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:13.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:13.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:13.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:13.002 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:13.002 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:13.002 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:13.002 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:13.002 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:13.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:13.021 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T22:22:13.021 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:13.021 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:13.021 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:13.021 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:13.021 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:13.021 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:13.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:13.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:22:13.080 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:13.089 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2026-02-25T22:22:13.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:13.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:13.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:13.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:13.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:22:13.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:22:13.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:13.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:13.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:13.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:13.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:22:13.092 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:13.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f71643ad-5946-449f-9f53-477d6aa1eb77 2026-02-25T22:22:13.093 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 f71643ad-5946-449f-9f53-477d6aa1eb77 2026-02-25T22:22:13.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f71643ad-5946-449f-9f53-477d6aa1eb77' 2026-02-25T22:22:13.094 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:13.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAVdp9pVMA3BhAAAYa1RSl+ya3qDdyy87DKZg== 2026-02-25T22:22:13.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAVdp9pVMA3BhAAAYa1RSl+ya3qDdyy87DKZg=="}' 2026-02-25T22:22:13.102 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f71643ad-5946-449f-9f53-477d6aa1eb77 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:13.211 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:13.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:13.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQAVdp9pVMA3BhAAAYa1RSl+ya3qDdyy87DKZg== --osd-uuid f71643ad-5946-449f-9f53-477d6aa1eb77 2026-02-25T22:22:13.236 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:13.238+0000 7f31143c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:13.237 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:13.238+0000 7f31143c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:13.238 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:13.238+0000 7f31143c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:13.239 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:13.238+0000 7f31143c0a40 -1 bdev(0x55eb3de76400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:13.239 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:13.238+0000 7f31143c0a40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:22:14.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:22:14.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:14.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:14.011 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:22:14.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:14.118 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:22:14.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:14.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2026-02-25T22:22:14.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:14.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:14.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:14.142 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:14.142+0000 7f713d004a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:14.144 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:14.146+0000 7f713d004a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:14.145 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:14.146+0000 7f713d004a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:14.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:14.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:14.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:14.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:14.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:14.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:14.224 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:14.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:14.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:14.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:14.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:14.557 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:14.558+0000 7f713d004a40 -1 Falling back to public interface 2026-02-25T22:22:15.267 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:15.270+0000 7f713d004a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:15.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:15.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:15.329 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:15.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:15.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:15.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:15.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:16.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:16.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:16.541 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:16.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:16.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:16.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:16.747 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3864629421,v1:127.0.0.1:6801/3864629421] [v2:127.0.0.1:6802/3864629421,v1:127.0.0.1:6803/3864629421] exists,up f71643ad-5946-449f-9f53-477d6aa1eb77 2026-02-25T22:22:16.748 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:16.748 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:16.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:16.749 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T22:22:16.749 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:16.749 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:16.750 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:16.750 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:16.750 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:16.750 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-osd.0.asok 2026-02-25T22:22:16.750 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:16.750 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-osd.0.asok log flush 2026-02-25T22:22:16.814 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:16.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:22:16.821 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:12.770+0000 7fd5cc0f1ec0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T22:22:16.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2026-02-25T22:22:16.822 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:14.558+0000 7f713d004a40 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T22:22:16.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:22:16.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:16.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:16.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:16.824 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:16.824 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:16.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:16.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:16.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:16.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:16.933 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:16.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:16.934 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:16.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:16.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:16.936 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:16.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:16.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:16.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:16.938 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:16.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:16.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:16.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:16.942 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:16.943 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:16.943 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:16.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:16.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:16.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:16.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T22:22:16.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T22:22:16.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:16.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:22:16.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:16.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:16.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:16.946 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:16.946 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:16.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:16.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:16.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:16.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:16.947 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:16.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:16.948 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:16.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:16.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:16.950 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:16.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:16.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:16.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:16.952 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:16.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:16.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:16.954 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:16.955 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:16.955 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:16.955 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:16.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:16.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:16.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:16.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:22:16.957 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:16.957 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:16.958 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:16.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:22:16.959 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:16.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:22:16.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:16.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:22:16.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2026-02-25T22:22:16.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:16.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:22:16.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:22:16.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:22:16.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:22:16.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2026-02-25T22:22:16.981 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:22:16.981 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:16.981 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:16.981 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:16.982 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:16.982 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:16.982 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:16.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2026-02-25T22:22:17.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:17.011 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:17.011 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:17.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:17.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:17.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:17.013 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:17.013 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:17.013 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:17.013 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:17.013 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.013 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:17.014 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:17.014 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:17.014 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:22:17.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:17.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:17.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:17.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:17.068 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:17.068 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:17.068 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:17.068 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:17.068 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:17.069 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.069 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:17.069 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:17.069 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:17.069 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:17.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:22:17.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:17.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:17.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:17.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:17.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:22:17.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:17.230 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:17.230 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:17.230 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:17.231 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:17.231 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.231 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:17.231 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:17.231 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:17.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:17.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T22:22:17.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:17.251 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:17.251 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:17.251 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.251 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:17.251 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:17.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:17.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:22:17.299 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:17.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2026-02-25T22:22:17.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:17.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:17.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:17.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:17.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:22:17.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:22:17.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:17.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:17.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:17.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2026-02-25T22:22:17.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:22:17.311 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:17.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2120d82e-ba85-4ad1-a663-e94796d081d4 2026-02-25T22:22:17.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2120d82e-ba85-4ad1-a663-e94796d081d4' 2026-02-25T22:22:17.312 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 2120d82e-ba85-4ad1-a663-e94796d081d4 2026-02-25T22:22:17.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:17.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAZdp9p82oxExAA4b3ETDPJ8r0AsDcg9l3InA== 2026-02-25T22:22:17.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAZdp9p82oxExAA4b3ETDPJ8r0AsDcg9l3InA=="}' 2026-02-25T22:22:17.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2120d82e-ba85-4ad1-a663-e94796d081d4 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:17.429 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:17.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:17.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQAZdp9p82oxExAA4b3ETDPJ8r0AsDcg9l3InA== --osd-uuid 2120d82e-ba85-4ad1-a663-e94796d081d4 2026-02-25T22:22:17.453 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:17.454+0000 7f68ab23da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:17.455 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:17.454+0000 7f68ab23da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:17.455 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:17.458+0000 7f68ab23da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:17.456 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:17.458+0000 7f68ab23da40 -1 bdev(0x558f79e66400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:17.456 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:17.458+0000 7f68ab23da40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:22:18.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:22:18.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:18.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:18.232 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:22:18.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:18.350 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:22:18.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:18.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2026-02-25T22:22:18.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:18.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:18.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:18.374 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:18.374+0000 7fc4fd0c3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:18.376 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:18.378+0000 7fc4fd0c3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:18.377 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:18.378+0000 7fc4fd0c3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:18.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:18.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:18.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:18.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:18.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:18.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:18.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:18.456 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:18.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:18.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:18.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:18.781 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:18.782+0000 7fc4fd0c3a40 -1 Falling back to public interface 2026-02-25T22:22:19.547 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:19.550+0000 7fc4fd0c3a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:19.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:19.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:19.561 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:19.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:19.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:19.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:19.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:20.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:20.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:20.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:20.772 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:20.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:20.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:20.981 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4146637217,v1:127.0.0.1:6801/4146637217] [v2:127.0.0.1:6802/4146637217,v1:127.0.0.1:6803/4146637217] exists,up 2120d82e-ba85-4ad1-a663-e94796d081d4 2026-02-25T22:22:20.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:20.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:20.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:20.982 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T22:22:20.982 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:20.982 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:20.984 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:20.984 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:20.984 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:20.984 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-osd.0.asok 2026-02-25T22:22:20.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:20.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-osd.0.asok log flush 2026-02-25T22:22:21.039 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:21.044 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:22:21.046 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:16.998+0000 7fef61fa2ec0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T22:22:21.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2026-02-25T22:22:21.047 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:18.782+0000 7fc4fd0c3a40 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T22:22:21.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:22:21.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:21.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:21.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:21.049 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:21.049 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:21.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:21.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:21.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:21.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:21.157 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:21.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:21.158 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:21.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:21.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:21.160 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:21.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:21.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:21.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:21.161 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:21.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:21.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:21.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:21.168 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:21.168 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:21.168 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:21.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:21.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:21.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:21.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:21.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:21.173 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:21.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:21.174 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:21.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:21.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:21.176 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:21.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:21.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:21.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:21.178 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:21.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:21.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:21.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:21.181 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:21.181 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:21.181 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:21.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:21.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:21.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:21.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:22:21.184 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:21.184 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:21.184 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:21.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:22:21.186 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:21.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:22:21.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:21.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:22:21.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2026-02-25T22:22:21.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:21.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:22:21.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:22:21.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:22:21.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:22:21.188 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2026-02-25T22:22:21.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:22:21.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:21.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:21.208 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:21.208 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:21.209 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:21.209 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:21.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2026-02-25T22:22:21.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:21.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:21.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:21.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:21.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:21.240 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:21.240 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:21.240 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:21.240 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:21.240 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:21.241 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:21.241 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:21.241 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:21.241 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:21.241 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:22:21.305 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:21.305 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:21.305 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:21.305 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:21.305 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:21.307 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:21.307 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:21.307 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:21.307 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:21.307 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:21.307 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:21.307 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:21.307 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:21.307 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:21.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:22:21.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:21.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:21.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:21.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:21.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:22:21.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:21.473 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:21.473 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:21.473 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:21.474 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:21.474 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:21.474 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:21.474 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:21.474 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:21.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:21.492 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T22:22:21.492 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:21.492 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:21.492 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:21.492 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:21.492 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:21.492 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:21.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:21.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:22:21.556 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:21.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2026-02-25T22:22:21.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:21.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:21.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:21.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:22:21.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:22:21.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:21.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:21.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:21.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:21.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:21.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2026-02-25T22:22:21.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:22:21.568 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:21.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e7f5f858-425b-4b9c-b2e3-6f4fd98df3cb 2026-02-25T22:22:21.568 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 e7f5f858-425b-4b9c-b2e3-6f4fd98df3cb 2026-02-25T22:22:21.569 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e7f5f858-425b-4b9c-b2e3-6f4fd98df3cb' 2026-02-25T22:22:21.569 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:21.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAddp9p2qGqIhAABryH/Tym2Yi755rbxWTo3w== 2026-02-25T22:22:21.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAddp9p2qGqIhAABryH/Tym2Yi755rbxWTo3w=="}' 2026-02-25T22:22:21.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e7f5f858-425b-4b9c-b2e3-6f4fd98df3cb -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:21.688 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:21.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:21.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQAddp9p2qGqIhAABryH/Tym2Yi755rbxWTo3w== --osd-uuid e7f5f858-425b-4b9c-b2e3-6f4fd98df3cb 2026-02-25T22:22:21.712 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:21.714+0000 7f795b19ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:21.714 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:21.714+0000 7f795b19ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:21.715 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:21.714+0000 7f795b19ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:21.715 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:21.718+0000 7f795b19ea40 -1 bdev(0x5583087e8400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:21.715 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:21.718+0000 7f795b19ea40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:22:22.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:22:22.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:22.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:22.496 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:22:22.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:22.615 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:22:22.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:22.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2026-02-25T22:22:22.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:22.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:22.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:22.639 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:22.642+0000 7f6cd4f8ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:22.641 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:22.642+0000 7f6cd4f8ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:22.643 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:22.642+0000 7f6cd4f8ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:22.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:22.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:22.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:22.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:22.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:22.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:22.723 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:22.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:22.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:22.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:22.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:23.081 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:23.082+0000 7f6cd4f8ba40 -1 Falling back to public interface 2026-02-25T22:22:23.805 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:23.806+0000 7f6cd4f8ba40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:23.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:23.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:23.828 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:23.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:23.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:23.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:24.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:25.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:25.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:25.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:25.042 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:25.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:25.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:25.248 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2373110893,v1:127.0.0.1:6801/2373110893] [v2:127.0.0.1:6802/2373110893,v1:127.0.0.1:6803/2373110893] exists,up e7f5f858-425b-4b9c-b2e3-6f4fd98df3cb 2026-02-25T22:22:25.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:25.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:25.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:25.249 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T22:22:25.249 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:25.249 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:25.250 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:25.250 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:25.250 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:25.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-osd.0.asok 2026-02-25T22:22:25.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:25.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-osd.0.asok log flush 2026-02-25T22:22:25.314 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:25.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:22:25.321 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:21.226+0000 7f8ad3083ec0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T22:22:25.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2026-02-25T22:22:25.323 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:23.082+0000 7f6cd4f8ba40 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T22:22:25.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:22:25.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:25.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:25.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:25.324 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:25.324 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:25.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:25.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:25.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:25.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:25.434 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:25.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:25.435 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:25.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:25.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:25.436 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:25.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:25.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:25.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:25.438 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:25.439 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:25.440 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:25.440 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:25.444 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:25.444 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:25.445 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:25.445 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:25.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:25.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:25.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T22:22:25.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T22:22:25.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:25.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:22:25.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:25.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:25.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:25.448 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:25.448 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:25.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:25.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:25.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:25.449 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:25.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:25.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:25.450 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:25.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:25.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:25.452 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:25.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:25.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:25.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:25.454 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:25.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:25.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:25.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:25.457 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:25.457 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:25.457 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:25.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:25.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:25.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:25.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:22:25.460 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:25.460 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:25.460 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:25.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:22:25.461 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:25.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:22:25.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:25.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:22:25.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2026-02-25T22:22:25.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:25.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:22:25.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:22:25.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:22:25.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:22:25.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2026-02-25T22:22:25.483 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:22:25.483 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:25.483 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:25.484 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:25.484 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:25.484 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:25.484 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:25.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2026-02-25T22:22:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:25.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:25.515 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:25.515 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:25.515 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:25.515 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:25.516 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:25.516 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:25.516 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:25.516 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:25.516 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:25.516 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:25.517 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:25.517 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:25.517 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:22:25.571 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:25.571 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:25.571 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:25.571 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:25.572 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:25.573 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:25.573 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:25.573 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:25.573 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:25.573 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:25.573 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:25.573 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:25.574 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:25.574 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:25.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:22:25.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:25.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:25.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:25.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:25.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:22:25.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:25.736 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:25.736 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:25.736 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:25.737 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:25.737 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:25.737 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:25.737 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:25.737 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:25.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:25.757 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T22:22:25.757 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:25.757 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:25.757 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:25.757 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:25.757 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:25.757 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:25.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:25.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:22:25.806 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:25.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2026-02-25T22:22:25.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:25.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:25.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:22:25.816 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:25.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:25.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:25.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:25.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:25.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2026-02-25T22:22:25.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:22:25.818 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:25.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a04c9872-2f35-40a5-aa54-e1cf6ef6f392 2026-02-25T22:22:25.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a04c9872-2f35-40a5-aa54-e1cf6ef6f392' 2026-02-25T22:22:25.819 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 a04c9872-2f35-40a5-aa54-e1cf6ef6f392 2026-02-25T22:22:25.819 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:25.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAhdp9pUsiGMRAAHCYhUpW9TjNYMp93x2HxjA== 2026-02-25T22:22:25.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAhdp9pUsiGMRAAHCYhUpW9TjNYMp93x2HxjA=="}' 2026-02-25T22:22:25.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a04c9872-2f35-40a5-aa54-e1cf6ef6f392 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:25.938 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:25.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:25.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQAhdp9pUsiGMRAAHCYhUpW9TjNYMp93x2HxjA== --osd-uuid a04c9872-2f35-40a5-aa54-e1cf6ef6f392 2026-02-25T22:22:25.962 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:25.962+0000 7f91b83b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:25.963 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:25.966+0000 7f91b83b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:25.964 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:25.966+0000 7f91b83b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:25.964 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:25.966+0000 7f91b83b9a40 -1 bdev(0x563bf8bbc400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:25.964 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:25.966+0000 7f91b83b9a40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:22:26.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:22:26.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:26.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:26.708 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:22:26.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:26.829 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:22:26.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:26.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2026-02-25T22:22:26.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:26.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:26.831 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:26.853 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:26.854+0000 7fa0ac21fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:26.855 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:26.854+0000 7fa0ac21fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:26.856 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:26.858+0000 7fa0ac21fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:26.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:26.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:26.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:26.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:26.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:26.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:26.935 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:26.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:26.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:26.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:27.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:27.253 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:27.254+0000 7fa0ac21fa40 -1 Falling back to public interface 2026-02-25T22:22:27.935 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:27.934+0000 7fa0ac21fa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:28.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:28.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:28.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:28.050 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:28.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:28.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:28.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:29.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:29.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:29.280 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:29.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:29.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:29.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:29.466 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/335973932,v1:127.0.0.1:6801/335973932] [v2:127.0.0.1:6802/335973932,v1:127.0.0.1:6803/335973932] exists,up a04c9872-2f35-40a5-aa54-e1cf6ef6f392 2026-02-25T22:22:29.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:29.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:29.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:29.467 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T22:22:29.468 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:29.468 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:29.469 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:29.469 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:29.469 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:29.469 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-osd.0.asok 2026-02-25T22:22:29.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:29.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-osd.0.asok log flush 2026-02-25T22:22:29.531 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:29.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:22:29.538 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:25.502+0000 7f6078d8bec0 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T22:22:29.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2026-02-25T22:22:29.539 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:27.254+0000 7fa0ac21fa40 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T22:22:29.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:22:29.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:29.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:29.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:29.540 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:29.540 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:29.649 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:29.649 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:29.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:29.651 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:29.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:29.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:29.652 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:29.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:29.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:29.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:29.654 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:29.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:29.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:29.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:29.659 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:29.659 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:29.659 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:29.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:29.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:29.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:29.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T22:22:29.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T22:22:29.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:29.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:22:29.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:29.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:29.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:29.662 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:29.663 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:29.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:29.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:29.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:29.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:29.664 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:29.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:29.665 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:29.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:29.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:29.667 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:29.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:29.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:29.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:29.668 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:29.669 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:29.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:29.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:29.671 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:29.671 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:29.671 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:29.672 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:29.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:29.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:29.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:22:29.674 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:29.674 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:29.674 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:29.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:22:29.676 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:29.677 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:22:29.677 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:29.677 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:22:29.677 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2026-02-25T22:22:29.677 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:29.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:22:29.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:22:29.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:22:29.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:22:29.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2026-02-25T22:22:29.698 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:22:29.698 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:29.698 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:29.698 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:29.698 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:29.698 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:29.699 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:29.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2026-02-25T22:22:29.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:29.728 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:29.728 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:29.728 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:29.729 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:29.730 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:29.730 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:29.730 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:29.730 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:29.730 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:29.730 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:29.730 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:29.730 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:29.731 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:29.731 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:22:29.785 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:29.785 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:29.785 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:29.785 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:29.786 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:29.786 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:29.786 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:29.786 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:29.786 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:29.787 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:29.787 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:29.787 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:29.787 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:29.787 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:29.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:22:29.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:29.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:29.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:29.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:29.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:22:29.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:29.948 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:29.948 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:29.948 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:29.949 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:29.949 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:29.949 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:29.949 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:29.950 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:29.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:29.969 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T22:22:29.969 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:29.969 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:29.969 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:29.969 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:29.969 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:29.969 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:29.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:29.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:22:30.017 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:30.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2026-02-25T22:22:30.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:30.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:30.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:30.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:30.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:22:30.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:22:30.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:30.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:30.026 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:30.027 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:30.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:30.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:30.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:30.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:30.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:22:30.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:22:30.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:30.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:30.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:30.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:30.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:30.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2026-02-25T22:22:30.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:22:30.029 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:30.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4f47cf2e-0f7f-46b3-b849-8c3ac5703770 2026-02-25T22:22:30.029 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 4f47cf2e-0f7f-46b3-b849-8c3ac5703770 2026-02-25T22:22:30.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4f47cf2e-0f7f-46b3-b849-8c3ac5703770' 2026-02-25T22:22:30.030 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:30.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAmdp9pD+qJAhAAAyeLEahNkg4LhL9dU6LBMg== 2026-02-25T22:22:30.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAmdp9pD+qJAhAAAyeLEahNkg4LhL9dU6LBMg=="}' 2026-02-25T22:22:30.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4f47cf2e-0f7f-46b3-b849-8c3ac5703770 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:30.150 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:30.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:30.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQAmdp9pD+qJAhAAAyeLEahNkg4LhL9dU6LBMg== --osd-uuid 4f47cf2e-0f7f-46b3-b849-8c3ac5703770 2026-02-25T22:22:30.173 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:30.174+0000 7fc677387a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:30.175 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:30.174+0000 7fc677387a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:30.176 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:30.178+0000 7fc677387a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:30.176 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:30.178+0000 7fc677387a40 -1 bdev(0x55f3963d2400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:30.176 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:30.178+0000 7fc677387a40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:22:30.986 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:22:30.986 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:30.987 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:22:30.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:30.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:31.106 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:22:31.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:31.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2026-02-25T22:22:31.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:31.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:31.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:31.130 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:31.130+0000 7f1adfc19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:31.132 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:31.134+0000 7f1adfc19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:31.134 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:31.134+0000 7f1adfc19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:31.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:31.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:31.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:31.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:31.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:31.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:31.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:31.212 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:31.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:31.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:31.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:31.621 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:31.622+0000 7f1adfc19a40 -1 Falling back to public interface 2026-02-25T22:22:32.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:32.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:32.323 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:32.323 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:32.323 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:32.323 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:32.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:32.632 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:32.634+0000 7f1adfc19a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:33.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:33.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:33.539 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:33.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:33.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:33.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:33.746 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1708997657,v1:127.0.0.1:6801/1708997657] [v2:127.0.0.1:6802/1708997657,v1:127.0.0.1:6803/1708997657] exists,up 4f47cf2e-0f7f-46b3-b849-8c3ac5703770 2026-02-25T22:22:33.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:33.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:33.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:33.747 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T22:22:33.747 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:33.748 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:33.748 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:33.748 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:33.749 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:33.749 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-osd.0.asok 2026-02-25T22:22:33.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:33.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-osd.0.asok log flush 2026-02-25T22:22:33.808 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:33.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:22:33.815 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:29.718+0000 7f6344054ec0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T22:22:33.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2026-02-25T22:22:33.816 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:31.622+0000 7f1adfc19a40 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T22:22:33.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:22:33.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:33.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:33.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:33.818 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:33.818 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:33.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:33.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:33.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:33.926 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:33.926 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:33.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:33.927 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:33.929 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:33.929 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:33.929 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:33.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:33.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:33.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:33.931 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:33.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:33.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:33.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:33.938 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:33.938 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:33.938 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:33.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:33.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:33.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:33.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T22:22:33.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T22:22:33.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:33.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T22:22:33.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:33.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:33.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:33.941 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:33.941 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:33.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:33.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:33.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:33.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:33.943 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:33.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:33.944 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:33.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:33.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:33.946 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:33.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:33.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:33.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:33.948 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:33.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:33.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:33.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:33.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:33.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:33.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:33.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:33.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:33.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:33.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T22:22:33.953 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:33.953 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:33.953 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:33.954 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11498 2026-02-25T22:22:33.955 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:33.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:22:33.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:33.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T22:22:33.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2026-02-25T22:22:33.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:33.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:22:33.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:22:33.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:22:33.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T22:22:33.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2026-02-25T22:22:33.978 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:22:33.978 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:33.978 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:33.978 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:33.978 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:33.979 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:33.979 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:33.979 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2026-02-25T22:22:34.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:34.006 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:34.006 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:34.006 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:34.006 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:34.007 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:34.008 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:34.008 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:34.008 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:34.008 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:34.008 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:34.008 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:34.008 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:34.009 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:34.009 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get fsid 2026-02-25T22:22:34.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:34.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:34.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:34.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:34.079 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:34.079 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:34.079 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:34.079 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:34.079 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:34.079 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:34.080 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:34.080 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:34.080 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:34.080 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11498/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:34.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T22:22:34.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:34.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:34.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:34.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:34.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T22:22:34.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:34.244 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:34.244 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:34.245 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:34.245 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:34.245 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:34.245 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:34.246 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:34.246 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:34.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:34.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T22:22:34.265 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:34.265 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:34.265 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:34.265 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:34.265 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:34.266 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-mon.a.asok 2026-02-25T22:22:34.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:34.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-mon.a.asok log flush 2026-02-25T22:22:34.313 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:34.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2026-02-25T22:22:34.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:34.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:34.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:34.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:34.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T22:22:34.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T22:22:34.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:34.322 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:34.323 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:34.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' 2026-02-25T22:22:34.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:34.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:34.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=330f661e-bedd-402a-991a-0ecf5e54266e 2026-02-25T22:22:34.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 330f661e-bedd-402a-991a-0ecf5e54266e' 2026-02-25T22:22:34.326 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 330f661e-bedd-402a-991a-0ecf5e54266e 2026-02-25T22:22:34.326 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:34.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAqdp9p/NkvFBAA+UoiKC5uZD20m2s5Qt7s2A== 2026-02-25T22:22:34.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAqdp9p/NkvFBAA+UoiKC5uZD20m2s5Qt7s2A=="}' 2026-02-25T22:22:34.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 330f661e-bedd-402a-991a-0ecf5e54266e -i td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:34.447 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:34.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T22:22:34.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQAqdp9p/NkvFBAA+UoiKC5uZD20m2s5Qt7s2A== --osd-uuid 330f661e-bedd-402a-991a-0ecf5e54266e 2026-02-25T22:22:34.472 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:34.474+0000 7fccd0789a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:34.474 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:34.474+0000 7fccd0789a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:34.475 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:34.474+0000 7fccd0789a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:34.475 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:34.478+0000 7fccd0789a40 -1 bdev(0x560e319f6400 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:34.475 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:34.478+0000 7fccd0789a40 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T22:22:35.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T22:22:35.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:35.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:35.169 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:22:35.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:35.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:35.287 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:22:35.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39ae6496-7b82-4f2c-a474-41df38ff89e5 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.11498/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2026-02-25T22:22:35.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:35.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:35.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:35.311 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:35.310+0000 7f146dae2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:35.313 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:35.314+0000 7f146dae2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:35.314 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:35.314+0000 7f146dae2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:35.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:35.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:35.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:35.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:35.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:35.392 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:35.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:35.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:35.725 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:35.726+0000 7f146dae2a40 -1 Falling back to public interface 2026-02-25T22:22:36.385 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:36.386+0000 7f146dae2a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:36.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:36.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:36.498 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:36.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:36.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:36.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:36.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:37.603 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:37.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:37.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:37.604 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:37.605 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:37.605 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:37.816 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/652884639,v1:127.0.0.1:6801/652884639] [v2:127.0.0.1:6802/652884639,v1:127.0.0.1:6803/652884639] exists,up 330f661e-bedd-402a-991a-0ecf5e54266e 2026-02-25T22:22:37.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:37.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:37.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:37.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T22:22:37.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:37.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:37.818 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:37.818 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:37.818 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:37.818 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11498/ceph-osd.0.asok 2026-02-25T22:22:37.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T22:22:37.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.11498/ceph-osd.0.asok log flush 2026-02-25T22:22:37.883 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:37.888 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-25T22:22:37.890 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:33.998+0000 7f93b727fec0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T22:22:37.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2026-02-25T22:22:37.891 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:35.726+0000 7f146dae2a40 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T22:22:37.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T22:22:37.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:37.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:37.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:37.892 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:37.892 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:37.893 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:37.893 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:37.893 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:38.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:38.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:38.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:38.002 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:38.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:38.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:38.004 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:38.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:38.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:38.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:38.005 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:38.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:38.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:38.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:38.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:38.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:38.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:38.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:38.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:38.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:38.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2026-02-25T22:22:38.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T22:22:38.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/test-erasure-code-plugins 0 2026-02-25T22:22:38.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T22:22:38.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T22:22:38.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T22:22:38.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:38.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:38.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:38.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:38.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:38.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:38.017 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:38.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:38.018 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:38.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:38.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:38.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:38.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:38.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:38.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:38.022 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:38.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:38.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T22:22:38.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T22:22:38.025 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:38.025 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:38.026 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11498 2026-02-25T22:22:38.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11498 2026-02-25T22:22:38.027 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:38.027 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:38.027 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T22:22:38.028 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T22:22:38.029 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T22:22:38.079 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2026-02-25T22:22:38.080 DEBUG:teuthology.orchestra.run.trial125:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61d4431d3ec879bd98659388b3655acf3d392418 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2026-02-25T22:22:38.133 INFO:tasks.workunit.client.0.trial125.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T22:22:38.136 INFO:tasks.workunit.client.0.trial125.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T22:22:38.136 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:22:38.136 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:22:38.136 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T22:22:38.136 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T22:22:38.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T22:22:38.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T22:22:38.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T22:22:38.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T22:22:38.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/test-erasure-code 2026-02-25T22:22:38.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2026-02-25T22:22:38.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2026-02-25T22:22:38.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2026-02-25T22:22:38.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2026-02-25T22:22:38.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2026-02-25T22:22:38.137 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2026-02-25T22:22:38.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none ' 2026-02-25T22:22:38.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:22:38.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2026-02-25T22:22:38.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code 2026-02-25T22:22:38.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code 2026-02-25T22:22:38.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2026-02-25T22:22:38.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:38.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2026-02-25T22:22:38.140 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:38.140 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:38.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:38.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:38.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:38.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:38.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:38.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:38.142 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:38.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:38.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:38.143 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:38.144 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:38.144 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:38.144 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:38.144 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:38.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:38.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:38.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2026-02-25T22:22:38.148 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:38.148 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:38.148 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:38.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.45907 2026-02-25T22:22:38.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:38.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:38.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code 2026-02-25T22:22:38.151 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:38.151 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:38.151 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:38.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.45907 2026-02-25T22:22:38.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:38.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T22:22:38.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T22:22:38.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:38.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2026-02-25T22:22:38.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2026-02-25T22:22:38.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code 2026-02-25T22:22:38.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:22:38.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:22:38.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:22:38.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code/a 2026-02-25T22:22:38.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2026-02-25T22:22:38.178 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:22:38.178 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:38.178 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:38.178 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:38.178 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:38.178 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:38.178 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:38.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:22:38.206 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:38.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:38.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:38.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:38.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:38.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:38.208 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:38.208 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:38.209 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:38.209 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:38.209 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:38.209 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:38.209 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.45907/ceph-mon.a.asok 2026-02-25T22:22:38.209 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:38.209 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.45907/ceph-mon.a.asok config get fsid 2026-02-25T22:22:38.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:38.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:38.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:38.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:38.266 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:38.266 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:38.266 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:38.266 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:38.266 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:38.266 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:38.266 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:38.266 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.45907/ceph-mon.a.asok 2026-02-25T22:22:38.266 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:38.266 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.45907/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:38.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2026-02-25T22:22:38.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code 2026-02-25T22:22:38.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:38.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:38.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:38.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code/x 2026-02-25T22:22:38.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:38.428 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:38.428 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:38.428 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:38.429 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:38.429 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:38.430 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:38.430 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:38.430 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:38.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:38.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2026-02-25T22:22:38.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:38.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:38.449 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:38.449 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:38.449 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:38.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.45907/ceph-mon.a.asok 2026-02-25T22:22:38.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2026-02-25T22:22:38.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.45907/ceph-mon.a.asok log flush 2026-02-25T22:22:38.498 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:22:38.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2026-02-25T22:22:38.506 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:38.198+0000 7f0141450ec0 0 load: jerasure load: lrc 2026-02-25T22:22:38.506 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2026-02-25T22:22:38.510 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T22:22:38.510 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2026-02-25T22:22:38.510 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T22:22:38.510 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:38.510 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:38.510 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:38.510 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/0 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:38.511 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:38.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:38.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:38.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:38.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:22:38.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/0 2026-02-25T22:22:38.513 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:38.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=efb58f4b-a795-470d-b144-b2330f04afbf 2026-02-25T22:22:38.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 efb58f4b-a795-470d-b144-b2330f04afbf' 2026-02-25T22:22:38.514 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 efb58f4b-a795-470d-b144-b2330f04afbf 2026-02-25T22:22:38.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:38.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAudp9ppCFfHxAAvdHf/z5zVSdnJ/MGxieptg== 2026-02-25T22:22:38.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAudp9ppCFfHxAAvdHf/z5zVSdnJ/MGxieptg=="}' 2026-02-25T22:22:38.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new efb58f4b-a795-470d-b144-b2330f04afbf -i td/test-erasure-code/0/new.json 2026-02-25T22:22:38.634 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:38.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/0/new.json 2026-02-25T22:22:38.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAudp9ppCFfHxAAvdHf/z5zVSdnJ/MGxieptg== --osd-uuid efb58f4b-a795-470d-b144-b2330f04afbf 2026-02-25T22:22:38.660 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:38.662+0000 7f719d25da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:38.661 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:38.662+0000 7f719d25da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:38.662 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:38.662+0000 7f719d25da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:38.663 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:38.662+0000 7f719d25da40 -1 bdev(0x559534b6a400 td/test-erasure-code/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:38.663 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:38.662+0000 7f719d25da40 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2026-02-25T22:22:39.379 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/0/keyring 2026-02-25T22:22:39.379 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:39.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:39.380 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:22:39.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:39.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:39.499 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:22:39.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:22:39.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:39.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:39.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:39.522 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:39.522+0000 7fcf0cdb1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:39.525 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:39.526+0000 7fcf0cdb1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:39.526 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:39.526+0000 7fcf0cdb1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:39.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:39.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:39.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:39.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:39.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:39.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:39.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:39.607 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:39.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:39.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:39.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:39.917 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:39.918+0000 7fcf0cdb1a40 -1 Falling back to public interface 2026-02-25T22:22:40.627 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:40.626+0000 7fcf0cdb1a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:40.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:40.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:40.713 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:40.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:40.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:40.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:40.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:41.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:41.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:41.932 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:41.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:41.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:41.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:42.136 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3681190371,v1:127.0.0.1:6801/3681190371] [v2:127.0.0.1:6802/3681190371,v1:127.0.0.1:6803/3681190371] exists,up efb58f4b-a795-470d-b144-b2330f04afbf 2026-02-25T22:22:42.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:42.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:42.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:42.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T22:22:42.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2026-02-25T22:22:42.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T22:22:42.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:42.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:22:42.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:42.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/1 2026-02-25T22:22:42.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:22:42.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:42.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:42.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:42.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:22:42.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/1 2026-02-25T22:22:42.143 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:42.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d355f62d-5e40-4152-a5c9-a1192b007eb2 2026-02-25T22:22:42.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d355f62d-5e40-4152-a5c9-a1192b007eb2' 2026-02-25T22:22:42.144 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 d355f62d-5e40-4152-a5c9-a1192b007eb2 2026-02-25T22:22:42.144 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:42.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAydp9pAQpVCRAAdD/9GjAers/XBGPUDvuqqQ== 2026-02-25T22:22:42.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAydp9pAQpVCRAAdD/9GjAers/XBGPUDvuqqQ=="}' 2026-02-25T22:22:42.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d355f62d-5e40-4152-a5c9-a1192b007eb2 -i td/test-erasure-code/1/new.json 2026-02-25T22:22:42.368 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:42.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/1/new.json 2026-02-25T22:22:42.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAydp9pAQpVCRAAdD/9GjAers/XBGPUDvuqqQ== --osd-uuid d355f62d-5e40-4152-a5c9-a1192b007eb2 2026-02-25T22:22:42.393 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:42.394+0000 7fa476df3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:42.395 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:42.394+0000 7fa476df3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:42.396 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:42.398+0000 7fa476df3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:42.396 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:42.398+0000 7fa476df3a40 -1 bdev(0x559cb67e2400 td/test-erasure-code/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:42.397 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:42.398+0000 7fa476df3a40 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2026-02-25T22:22:43.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/1/keyring 2026-02-25T22:22:43.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:43.051 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:22:43.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:22:43.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:43.336 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:22:43.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:22:43.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:22:43.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:43.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:43.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:43.358 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:43.358+0000 7ff855c08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:43.360 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:43.362+0000 7ff855c08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:43.361 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:43.362+0000 7ff855c08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:43.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:22:43.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:43.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:22:43.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:43.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:43.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:43.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:43.551 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:43.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:43.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:22:43.717 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:43.718+0000 7ff855c08a40 -1 Falling back to public interface 2026-02-25T22:22:43.765 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:44.301 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:44.302+0000 7ff855c08a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:22:44.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:44.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:44.767 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:44.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:44.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:44.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:22:44.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:45.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:45.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:45.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:45.975 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:45.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:45.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:22:46.186 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1750226941,v1:127.0.0.1:6811/1750226941] [v2:127.0.0.1:6812/1750226941,v1:127.0.0.1:6813/1750226941] exists,up d355f62d-5e40-4152-a5c9-a1192b007eb2 2026-02-25T22:22:46.189 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:46.189 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:46.189 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:46.189 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T22:22:46.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2026-02-25T22:22:46.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T22:22:46.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:46.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:22:46.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:46.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/2 2026-02-25T22:22:46.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:22:46.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:46.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:46.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:46.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:46.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:46.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:46.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:46.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:22:46.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:22:46.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:46.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:46.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:46.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:46.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:46.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:22:46.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/2 2026-02-25T22:22:46.193 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:46.193 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=addd79b2-5cff-4c6b-813c-d233cc949408 2026-02-25T22:22:46.193 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 addd79b2-5cff-4c6b-813c-d233cc949408' 2026-02-25T22:22:46.193 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 addd79b2-5cff-4c6b-813c-d233cc949408 2026-02-25T22:22:46.194 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:46.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA2dp9pRnQ7DBAA5C8Iv3EULyM6fjlUpUzR9Q== 2026-02-25T22:22:46.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA2dp9pRnQ7DBAA5C8Iv3EULyM6fjlUpUzR9Q=="}' 2026-02-25T22:22:46.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new addd79b2-5cff-4c6b-813c-d233cc949408 -i td/test-erasure-code/2/new.json 2026-02-25T22:22:46.412 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:46.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/2/new.json 2026-02-25T22:22:46.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2dp9pRnQ7DBAA5C8Iv3EULyM6fjlUpUzR9Q== --osd-uuid addd79b2-5cff-4c6b-813c-d233cc949408 2026-02-25T22:22:46.437 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:46.438+0000 7f626b4d4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:46.438 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:46.438+0000 7f626b4d4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:46.439 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:46.442+0000 7f626b4d4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:46.440 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:46.442+0000 7f626b4d4a40 -1 bdev(0x563962630400 td/test-erasure-code/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:46.440 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:46.442+0000 7f626b4d4a40 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2026-02-25T22:22:47.098 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/2/keyring 2026-02-25T22:22:47.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:47.100 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:22:47.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:22:47.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:47.383 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:22:47.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:22:47.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:22:47.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:47.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:47.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:47.405 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:47.406+0000 7fb074810a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:47.407 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:47.410+0000 7fb074810a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:47.408 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:47.410+0000 7fb074810a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:47.594 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:22:47.594 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:47.594 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:22:47.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:47.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:47.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:47.595 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:47.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:47.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:47.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:22:47.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:47.809 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:47.810+0000 7fb074810a40 -1 Falling back to public interface 2026-02-25T22:22:48.389 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:48.390+0000 7fb074810a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:22:48.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:48.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:48.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:48.802 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:48.802 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:48.802 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:22:49.009 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3344274032,v1:127.0.0.1:6819/3344274032] [v2:127.0.0.1:6820/3344274032,v1:127.0.0.1:6821/3344274032] exists,up addd79b2-5cff-4c6b-813c-d233cc949408 2026-02-25T22:22:49.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:49.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/3 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:49.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:22:49.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/3 2026-02-25T22:22:49.016 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:49.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c248833a-f8a8-4136-a10f-63738620755b 2026-02-25T22:22:49.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 c248833a-f8a8-4136-a10f-63738620755b' 2026-02-25T22:22:49.017 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 c248833a-f8a8-4136-a10f-63738620755b 2026-02-25T22:22:49.017 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:49.027 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA5dp9p7STGARAACsiJAzU4F8xcN0f58ZYU/Q== 2026-02-25T22:22:49.027 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA5dp9p7STGARAACsiJAzU4F8xcN0f58ZYU/Q=="}' 2026-02-25T22:22:49.027 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c248833a-f8a8-4136-a10f-63738620755b -i td/test-erasure-code/3/new.json 2026-02-25T22:22:49.242 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:22:49.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/3/new.json 2026-02-25T22:22:49.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5dp9p7STGARAACsiJAzU4F8xcN0f58ZYU/Q== --osd-uuid c248833a-f8a8-4136-a10f-63738620755b 2026-02-25T22:22:49.268 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:49.270+0000 7fdb3ec95a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:49.269 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:49.270+0000 7fdb3ec95a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:49.271 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:49.270+0000 7fdb3ec95a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:49.271 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:49.274+0000 7fdb3ec95a40 -1 bdev(0x5600b3288400 td/test-erasure-code/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:49.271 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:49.274+0000 7fdb3ec95a40 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2026-02-25T22:22:49.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/3/keyring 2026-02-25T22:22:49.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:49.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:22:49.867 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:22:49.867 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:50.161 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:22:50.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:22:50.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:22:50.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:50.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:50.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:50.176 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:50.178+0000 7f38ab976a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:50.178 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:50.178+0000 7f38ab976a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:50.179 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:50.178+0000 7f38ab976a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:50.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:22:50.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:50.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:22:50.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:50.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:50.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:50.377 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:50.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:50.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:50.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:22:50.590 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:50.645 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:50.646+0000 7f38ab976a40 -1 Falling back to public interface 2026-02-25T22:22:51.221 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:51.222+0000 7f38ab976a40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:22:51.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:51.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:51.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:51.593 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:51.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:51.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:22:51.807 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:52.808 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:52.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:52.809 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:52.810 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:52.810 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:52.810 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:22:53.015 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/376704620,v1:127.0.0.1:6827/376704620] [v2:127.0.0.1:6828/376704620,v1:127.0.0.1:6829/376704620] exists,up c248833a-f8a8-4136-a10f-63738620755b 2026-02-25T22:22:53.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:53.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:53.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:53.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T22:22:53.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2026-02-25T22:22:53.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T22:22:53.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:53.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T22:22:53.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:53.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/4 2026-02-25T22:22:53.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:22:53.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:53.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:53.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:53.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2026-02-25T22:22:53.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:53.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:53.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:53.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:22:53.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:22:53.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:53.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:53.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:53.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:53.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:53.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:22:53.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/4 2026-02-25T22:22:53.022 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:53.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=add511f0-1f5b-40d0-84b5-63d895026bae 2026-02-25T22:22:53.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 add511f0-1f5b-40d0-84b5-63d895026bae' 2026-02-25T22:22:53.023 INFO:tasks.workunit.client.0.trial125.stdout:add osd4 add511f0-1f5b-40d0-84b5-63d895026bae 2026-02-25T22:22:53.023 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:53.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA9dp9p4jYHAhAAeWDHJyt8VV5yhsbNAk6ntw== 2026-02-25T22:22:53.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA9dp9p4jYHAhAAeWDHJyt8VV5yhsbNAk6ntw=="}' 2026-02-25T22:22:53.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new add511f0-1f5b-40d0-84b5-63d895026bae -i td/test-erasure-code/4/new.json 2026-02-25T22:22:53.249 INFO:tasks.workunit.client.0.trial125.stdout:4 2026-02-25T22:22:53.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/4/new.json 2026-02-25T22:22:53.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9dp9p4jYHAhAAeWDHJyt8VV5yhsbNAk6ntw== --osd-uuid add511f0-1f5b-40d0-84b5-63d895026bae 2026-02-25T22:22:53.273 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:53.274+0000 7f44962cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:53.275 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:53.274+0000 7f44962cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:53.276 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:53.278+0000 7f44962cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:53.276 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:53.278+0000 7f44962cba40 -1 bdev(0x55ce9d022400 td/test-erasure-code/4/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:53.276 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:53.278+0000 7f44962cba40 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2026-02-25T22:22:53.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/4/keyring 2026-02-25T22:22:53.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:53.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T22:22:53.910 INFO:tasks.workunit.client.0.trial125.stdout:adding osd4 key to auth repository 2026-02-25T22:22:53.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:54.193 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T22:22:54.194 INFO:tasks.workunit.client.0.trial125.stdout:start osd.4 2026-02-25T22:22:54.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:22:54.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:54.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:54.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:54.216 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:54.218+0000 7fe5c1a6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:54.218 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:54.218+0000 7fe5c1a6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:54.219 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:54.222+0000 7fe5c1a6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:54.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T22:22:54.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:54.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T22:22:54.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:54.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:54.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:54.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:54.409 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:54.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:54.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:22:54.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:54.669 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:54.670+0000 7fe5c1a6da40 -1 Falling back to public interface 2026-02-25T22:22:55.302 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:55.302+0000 7fe5c1a6da40 -1 osd.4 0 log_to_monitors true 2026-02-25T22:22:55.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:55.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:55.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:55.618 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:55.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:55.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:22:55.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:56.831 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:56.831 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:56.831 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:22:56.832 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:56.832 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:56.832 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:22:57.034 INFO:tasks.workunit.client.0.trial125.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3777636678,v1:127.0.0.1:6835/3777636678] [v2:127.0.0.1:6836/3777636678,v1:127.0.0.1:6837/3777636678] exists,up add511f0-1f5b-40d0-84b5-63d895026bae 2026-02-25T22:22:57.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:57.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:57.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:57.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T22:22:57.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2026-02-25T22:22:57.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T22:22:57.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:57.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T22:22:57.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:57.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/5 2026-02-25T22:22:57.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:57.039 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:57.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:57.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:57.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:22:57.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/5 2026-02-25T22:22:57.041 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:57.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c1450571-a937-48d3-b9df-6988c9294fc6 2026-02-25T22:22:57.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 c1450571-a937-48d3-b9df-6988c9294fc6' 2026-02-25T22:22:57.042 INFO:tasks.workunit.client.0.trial125.stdout:add osd5 c1450571-a937-48d3-b9df-6988c9294fc6 2026-02-25T22:22:57.042 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:57.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBdp9pedGEAxAAet1vXlVSXNNp50a+9pPr2Q== 2026-02-25T22:22:57.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBdp9pedGEAxAAet1vXlVSXNNp50a+9pPr2Q=="}' 2026-02-25T22:22:57.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c1450571-a937-48d3-b9df-6988c9294fc6 -i td/test-erasure-code/5/new.json 2026-02-25T22:22:57.277 INFO:tasks.workunit.client.0.trial125.stdout:5 2026-02-25T22:22:57.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/5/new.json 2026-02-25T22:22:57.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBdp9pedGEAxAAet1vXlVSXNNp50a+9pPr2Q== --osd-uuid c1450571-a937-48d3-b9df-6988c9294fc6 2026-02-25T22:22:57.313 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:57.313+0000 7fedd52bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:57.314 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:57.313+0000 7fedd52bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:57.315 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:57.317+0000 7fedd52bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:57.316 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:57.317+0000 7fedd52bea40 -1 bdev(0x558c09a58400 td/test-erasure-code/5/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:57.316 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:57.317+0000 7fedd52bea40 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2026-02-25T22:22:57.959 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/5/keyring 2026-02-25T22:22:57.959 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:57.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T22:22:57.960 INFO:tasks.workunit.client.0.trial125.stdout:adding osd5 key to auth repository 2026-02-25T22:22:57.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:58.245 INFO:tasks.workunit.client.0.trial125.stdout:start osd.5 2026-02-25T22:22:58.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T22:22:58.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:22:58.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:58.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:58.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:58.267 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:58.265+0000 7fd3360d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:58.268 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:58.269+0000 7fd3360d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:58.270 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:58.269+0000 7fd3360d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:58.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T22:22:58.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:58.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T22:22:58.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:58.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:58.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:58.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:58.457 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:22:58.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:58.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:22:58.653 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:58.653+0000 7fd3360d7a40 -1 Falling back to public interface 2026-02-25T22:22:58.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:59.313 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:22:59.313+0000 7fd3360d7a40 -1 osd.5 0 log_to_monitors true 2026-02-25T22:22:59.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:59.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:59.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:59.675 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:22:59.675 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:59.675 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:22:59.884 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:00.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:00.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:00.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:23:00.887 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:23:00.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:00.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:23:01.090 INFO:tasks.workunit.client.0.trial125.stdout:osd.5 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2995651127,v1:127.0.0.1:6843/2995651127] [v2:127.0.0.1:6844/2995651127,v1:127.0.0.1:6845/2995651127] exists,up c1450571-a937-48d3-b9df-6988c9294fc6 2026-02-25T22:23:01.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:23:01.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:23:01.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:23:01.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T22:23:01.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2026-02-25T22:23:01.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/6 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:23:01.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:23:01.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:23:01.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:23:01.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:23:01.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:23:01.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:23:01.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:23:01.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:23:01.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:23:01.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/6 2026-02-25T22:23:01.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:23:01.097 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5184d0a8-0efa-4a82-85e6-850f0226d9db 2026-02-25T22:23:01.097 INFO:tasks.workunit.client.0.trial125.stdout:add osd6 5184d0a8-0efa-4a82-85e6-850f0226d9db 2026-02-25T22:23:01.098 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 5184d0a8-0efa-4a82-85e6-850f0226d9db' 2026-02-25T22:23:01.098 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:23:01.106 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBFdp9p+x2ABhAA9zgrF7bd+AcPt1UJfTQVxg== 2026-02-25T22:23:01.106 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBFdp9p+x2ABhAA9zgrF7bd+AcPt1UJfTQVxg=="}' 2026-02-25T22:23:01.106 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5184d0a8-0efa-4a82-85e6-850f0226d9db -i td/test-erasure-code/6/new.json 2026-02-25T22:23:01.320 INFO:tasks.workunit.client.0.trial125.stdout:6 2026-02-25T22:23:01.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/6/new.json 2026-02-25T22:23:01.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFdp9p+x2ABhAA9zgrF7bd+AcPt1UJfTQVxg== --osd-uuid 5184d0a8-0efa-4a82-85e6-850f0226d9db 2026-02-25T22:23:01.345 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:01.345+0000 7fa9bdc7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:01.346 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:01.345+0000 7fa9bdc7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:01.347 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:01.349+0000 7fa9bdc7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:01.348 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:01.349+0000 7fa9bdc7aa40 -1 bdev(0x56544445e400 td/test-erasure-code/6/block) open stat got: (1) Operation not permitted 2026-02-25T22:23:01.348 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:01.349+0000 7fa9bdc7aa40 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2026-02-25T22:23:01.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/6/keyring 2026-02-25T22:23:01.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:23:01.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T22:23:01.976 INFO:tasks.workunit.client.0.trial125.stdout:adding osd6 key to auth repository 2026-02-25T22:23:01.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:23:02.264 INFO:tasks.workunit.client.0.trial125.stdout:start osd.6 2026-02-25T22:23:02.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T22:23:02.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:23:02.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:23:02.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:23:02.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:23:02.286 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:02.285+0000 7f1031e72a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:02.288 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:02.289+0000 7f1031e72a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:02.289 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:02.289+0000 7f1031e72a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:02.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T22:23:02.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:23:02.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T22:23:02.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:23:02.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:23:02.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:02.476 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:23:02.476 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:23:02.476 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:02.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:23:02.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:02.753 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:02.753+0000 7f1031e72a40 -1 Falling back to public interface 2026-02-25T22:23:03.382 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:03.381+0000 7f1031e72a40 -1 osd.6 0 log_to_monitors true 2026-02-25T22:23:03.689 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:03.689 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:03.689 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:23:03.690 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:23:03.690 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:03.690 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:23:03.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:04.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:04.902 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:04.902 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:23:04.902 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:23:04.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:04.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:23:05.112 INFO:tasks.workunit.client.0.trial125.stdout:osd.6 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/865235496,v1:127.0.0.1:6851/865235496] [v2:127.0.0.1:6852/865235496,v1:127.0.0.1:6853/865235496] exists,up 5184d0a8-0efa-4a82-85e6-850f0226d9db 2026-02-25T22:23:05.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:23:05.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:23:05.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:23:05.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T22:23:05.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2026-02-25T22:23:05.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T22:23:05.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:23:05.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-25T22:23:05.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:23:05.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/7 2026-02-25T22:23:05.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:23:05.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:23:05.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:23:05.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:23:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/7 2026-02-25T22:23:05.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:23:05.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9bb3846c-1feb-47af-959a-65a48e498863 2026-02-25T22:23:05.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 9bb3846c-1feb-47af-959a-65a48e498863' 2026-02-25T22:23:05.120 INFO:tasks.workunit.client.0.trial125.stdout:add osd7 9bb3846c-1feb-47af-959a-65a48e498863 2026-02-25T22:23:05.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:23:05.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBJdp9phUzbBxAALx8dpnNm2O65OvIgdb3Irw== 2026-02-25T22:23:05.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBJdp9phUzbBxAALx8dpnNm2O65OvIgdb3Irw=="}' 2026-02-25T22:23:05.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9bb3846c-1feb-47af-959a-65a48e498863 -i td/test-erasure-code/7/new.json 2026-02-25T22:23:05.342 INFO:tasks.workunit.client.0.trial125.stdout:7 2026-02-25T22:23:05.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/7/new.json 2026-02-25T22:23:05.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJdp9phUzbBxAALx8dpnNm2O65OvIgdb3Irw== --osd-uuid 9bb3846c-1feb-47af-959a-65a48e498863 2026-02-25T22:23:05.368 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:05.369+0000 7fb15fba2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:05.369 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:05.369+0000 7fb15fba2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:05.370 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:05.369+0000 7fb15fba2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:05.371 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:05.369+0000 7fb15fba2a40 -1 bdev(0x55a82e072400 td/test-erasure-code/7/block) open stat got: (1) Operation not permitted 2026-02-25T22:23:05.371 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:05.369+0000 7fb15fba2a40 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2026-02-25T22:23:06.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/7/keyring 2026-02-25T22:23:06.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:23:06.029 INFO:tasks.workunit.client.0.trial125.stdout:adding osd7 key to auth repository 2026-02-25T22:23:06.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-25T22:23:06.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:23:06.319 INFO:tasks.workunit.client.0.trial125.stdout:start osd.7 2026-02-25T22:23:06.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-25T22:23:06.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:23:06.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:23:06.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:23:06.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:23:06.343 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:06.341+0000 7f8507904a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:06.345 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:06.345+0000 7f8507904a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:06.346 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:06.345+0000 7f8507904a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:06.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-25T22:23:06.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:23:06.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-25T22:23:06.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:23:06.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:23:06.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:06.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:23:06.526 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:23:06.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:06.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-25T22:23:06.734 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:06.841 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:06.841+0000 7f8507904a40 -1 Falling back to public interface 2026-02-25T22:23:07.429 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:07.429+0000 7f8507904a40 -1 osd.7 0 log_to_monitors true 2026-02-25T22:23:07.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:07.736 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:23:07.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:07.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:23:07.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:07.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-25T22:23:07.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:08.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:08.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:08.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:23:08.947 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:23:08.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:08.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-25T22:23:09.156 INFO:tasks.workunit.client.0.trial125.stdout:osd.7 up in weight 1 up_from 36 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2687687546,v1:127.0.0.1:6859/2687687546] [v2:127.0.0.1:6860/2687687546,v1:127.0.0.1:6861/2687687546] exists,up 9bb3846c-1feb-47af-959a-65a48e498863 2026-02-25T22:23:09.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:23:09.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:23:09.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:23:09.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T22:23:09.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2026-02-25T22:23:09.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T22:23:09.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:23:09.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/8 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:23:09.160 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:23:09.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:23:09.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:23:09.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:23:09.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:23:09.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:23:09.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:23:09.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/8 2026-02-25T22:23:09.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:23:09.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b616c718-2be7-4af1-bddf-96b297352313 2026-02-25T22:23:09.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 b616c718-2be7-4af1-bddf-96b297352313' 2026-02-25T22:23:09.163 INFO:tasks.workunit.client.0.trial125.stdout:add osd8 b616c718-2be7-4af1-bddf-96b297352313 2026-02-25T22:23:09.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:23:09.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBNdp9pKyaMChAAquGew/oFx/Z5M4A69NlUzA== 2026-02-25T22:23:09.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBNdp9pKyaMChAAquGew/oFx/Z5M4A69NlUzA=="}' 2026-02-25T22:23:09.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b616c718-2be7-4af1-bddf-96b297352313 -i td/test-erasure-code/8/new.json 2026-02-25T22:23:09.388 INFO:tasks.workunit.client.0.trial125.stdout:8 2026-02-25T22:23:09.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/8/new.json 2026-02-25T22:23:09.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNdp9pKyaMChAAquGew/oFx/Z5M4A69NlUzA== --osd-uuid b616c718-2be7-4af1-bddf-96b297352313 2026-02-25T22:23:09.413 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:09.413+0000 7f99427eda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:09.414 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:09.413+0000 7f99427eda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:09.415 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:09.417+0000 7f99427eda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:09.416 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:09.417+0000 7f99427eda40 -1 bdev(0x55a45201e400 td/test-erasure-code/8/block) open stat got: (1) Operation not permitted 2026-02-25T22:23:09.416 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:09.417+0000 7f99427eda40 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2026-02-25T22:23:10.126 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/8/keyring 2026-02-25T22:23:10.126 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:23:10.127 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-25T22:23:10.127 INFO:tasks.workunit.client.0.trial125.stdout:adding osd8 key to auth repository 2026-02-25T22:23:10.127 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:23:10.402 INFO:tasks.workunit.client.0.trial125.stdout:start osd.8 2026-02-25T22:23:10.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-25T22:23:10.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:23:10.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:23:10.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:23:10.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:23:10.426 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:10.425+0000 7f8bbe591a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:10.428 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:10.429+0000 7f8bbe591a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:10.429 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:10.429+0000 7f8bbe591a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:10.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-25T22:23:10.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:23:10.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-25T22:23:10.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:23:10.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:23:10.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:10.619 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:23:10.619 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:23:10.619 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:10.620 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-25T22:23:10.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:10.884 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:10.885+0000 7f8bbe591a40 -1 Falling back to public interface 2026-02-25T22:23:11.532 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:11.533+0000 7f8bbe591a40 -1 osd.8 0 log_to_monitors true 2026-02-25T22:23:11.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:11.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:11.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:23:11.829 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:23:11.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:11.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-25T22:23:12.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:13.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:13.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:13.050 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:23:13.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:23:13.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:13.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-25T22:23:13.257 INFO:tasks.workunit.client.0.trial125.stdout:osd.8 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3418554784,v1:127.0.0.1:6867/3418554784] [v2:127.0.0.1:6868/3418554784,v1:127.0.0.1:6869/3418554784] exists,up b616c718-2be7-4af1-bddf-96b297352313 2026-02-25T22:23:13.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:23:13.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:23:13.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:23:13.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T22:23:13.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2026-02-25T22:23:13.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T22:23:13.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:23:13.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/9 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:23:13.261 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:23:13.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:23:13.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:23:13.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:23:13.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:23:13.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:23:13.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:23:13.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:23:13.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/9 2026-02-25T22:23:13.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:23:13.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0d0c8d03-6881-4e3b-9477-1d0d8e85e7b0 2026-02-25T22:23:13.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 0d0c8d03-6881-4e3b-9477-1d0d8e85e7b0' 2026-02-25T22:23:13.264 INFO:tasks.workunit.client.0.trial125.stdout:add osd9 0d0c8d03-6881-4e3b-9477-1d0d8e85e7b0 2026-02-25T22:23:13.265 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:23:13.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBRdp9pEoiGEBAA7QXnYcwytivh/2Ai8beUBA== 2026-02-25T22:23:13.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBRdp9pEoiGEBAA7QXnYcwytivh/2Ai8beUBA=="}' 2026-02-25T22:23:13.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0d0c8d03-6881-4e3b-9477-1d0d8e85e7b0 -i td/test-erasure-code/9/new.json 2026-02-25T22:23:13.492 INFO:tasks.workunit.client.0.trial125.stdout:9 2026-02-25T22:23:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/9/new.json 2026-02-25T22:23:13.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBRdp9pEoiGEBAA7QXnYcwytivh/2Ai8beUBA== --osd-uuid 0d0c8d03-6881-4e3b-9477-1d0d8e85e7b0 2026-02-25T22:23:13.517 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:13.517+0000 7f4f5c768a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:13.519 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:13.517+0000 7f4f5c768a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:13.520 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:13.521+0000 7f4f5c768a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:13.520 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:13.521+0000 7f4f5c768a40 -1 bdev(0x5574f0f3e400 td/test-erasure-code/9/block) open stat got: (1) Operation not permitted 2026-02-25T22:23:13.521 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:13.521+0000 7f4f5c768a40 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2026-02-25T22:23:14.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/9/keyring 2026-02-25T22:23:14.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:23:14.171 INFO:tasks.workunit.client.0.trial125.stdout:adding osd9 key to auth repository 2026-02-25T22:23:14.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2026-02-25T22:23:14.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:23:14.450 INFO:tasks.workunit.client.0.trial125.stdout:start osd.9 2026-02-25T22:23:14.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2026-02-25T22:23:14.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:23:14.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:23:14.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:23:14.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:23:14.473 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:14.473+0000 7f2e50935a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:14.474 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:14.473+0000 7f2e50935a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:14.475 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:14.477+0000 7f2e50935a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:14.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2026-02-25T22:23:14.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:23:14.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-25T22:23:14.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:23:14.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:23:14.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:14.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:23:14.667 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:23:14.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:14.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-25T22:23:14.829 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:14.829+0000 7f2e50935a40 -1 Falling back to public interface 2026-02-25T22:23:14.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:15.432 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:15.433+0000 7f2e50935a40 -1 osd.9 0 log_to_monitors true 2026-02-25T22:23:15.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:15.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:15.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:23:15.879 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:23:15.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:15.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-25T22:23:16.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:17.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:17.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:17.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:23:17.089 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:23:17.089 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:17.089 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-25T22:23:17.297 INFO:tasks.workunit.client.0.trial125.stdout:osd.9 up in weight 1 up_from 44 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/3760288217,v1:127.0.0.1:6875/3760288217] [v2:127.0.0.1:6876/3760288217,v1:127.0.0.1:6877/3760288217] exists,up 0d0c8d03-6881-4e3b-9477-1d0d8e85e7b0 2026-02-25T22:23:17.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:23:17.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:23:17.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:23:17.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=10 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/10 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2026-02-25T22:23:17.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:23:17.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:23:17.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:23:17.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:23:17.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:23:17.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:23:17.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:23:17.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:23:17.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:23:17.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:23:17.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:23:17.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/10 2026-02-25T22:23:17.303 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:23:17.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fb1bb5d4-ae70-423f-b130-1072bfe840a8 2026-02-25T22:23:17.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd10 fb1bb5d4-ae70-423f-b130-1072bfe840a8' 2026-02-25T22:23:17.304 INFO:tasks.workunit.client.0.trial125.stdout:add osd10 fb1bb5d4-ae70-423f-b130-1072bfe840a8 2026-02-25T22:23:17.305 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:23:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBVdp9pzPbiEhAAprdoD4FJZaosvBGgbzutwg== 2026-02-25T22:23:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBVdp9pzPbiEhAAprdoD4FJZaosvBGgbzutwg=="}' 2026-02-25T22:23:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fb1bb5d4-ae70-423f-b130-1072bfe840a8 -i td/test-erasure-code/10/new.json 2026-02-25T22:23:17.536 INFO:tasks.workunit.client.0.trial125.stdout:10 2026-02-25T22:23:17.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/10/new.json 2026-02-25T22:23:17.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 10 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVdp9pzPbiEhAAprdoD4FJZaosvBGgbzutwg== --osd-uuid fb1bb5d4-ae70-423f-b130-1072bfe840a8 2026-02-25T22:23:17.561 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:17.561+0000 7f5a0aba2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:17.562 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:17.561+0000 7f5a0aba2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:17.563 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:17.561+0000 7f5a0aba2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:17.563 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:17.565+0000 7f5a0aba2a40 -1 bdev(0x55a83b174400 td/test-erasure-code/10/block) open stat got: (1) Operation not permitted 2026-02-25T22:23:17.564 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:17.565+0000 7f5a0aba2a40 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2026-02-25T22:23:18.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/10/keyring 2026-02-25T22:23:18.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:23:18.227 INFO:tasks.workunit.client.0.trial125.stdout:adding osd10 key to auth repository 2026-02-25T22:23:18.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd10 key to auth repository 2026-02-25T22:23:18.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:23:18.510 INFO:tasks.workunit.client.0.trial125.stdout:start osd.10 2026-02-25T22:23:18.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.10 2026-02-25T22:23:18.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 10 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:23:18.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:23:18.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:23:18.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:23:18.535 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:18.533+0000 7f6c478b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:18.537 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:18.537+0000 7f6c478b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:18.538 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:18.537+0000 7f6c478b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:18.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 10 2026-02-25T22:23:18.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:23:18.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=10 2026-02-25T22:23:18.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:23:18.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:23:18.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:18.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:23:18.727 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:23:18.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:18.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2026-02-25T22:23:18.921 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:18.921+0000 7f6c478b9a40 -1 Falling back to public interface 2026-02-25T22:23:18.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:19.632 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:19.633+0000 7f6c478b9a40 -1 osd.10 0 log_to_monitors true 2026-02-25T22:23:19.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:19.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:19.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:23:19.937 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:23:19.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:19.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2026-02-25T22:23:20.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:21.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:21.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:21.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:23:21.153 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:23:21.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:21.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2026-02-25T22:23:21.358 INFO:tasks.workunit.client.0.trial125.stdout:osd.10 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/3927201260,v1:127.0.0.1:6883/3927201260] [v2:127.0.0.1:6884/3927201260,v1:127.0.0.1:6885/3927201260] exists,up fb1bb5d4-ae70-423f-b130-1072bfe840a8 2026-02-25T22:23:21.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:23:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:23:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:23:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2026-02-25T22:23:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:23:21.555 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' does not exist 2026-02-25T22:23:21.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:23:21.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:23:21.805 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:23:21.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:23:22.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:23:23.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2026-02-25T22:23:23.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:23:23.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:23:23.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:23:23.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:23:23.200 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:23:23.200 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:23:23.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:23:23.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:23:23.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:23:23.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:23:23.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:23:23.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:23:23.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:23:23.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:23:23.302 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:23:23.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:23:23.513 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:23:23.513 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:23:23.513 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:23:23.513 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:23:23.513 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:23:23.513 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:23:23.513 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:23:23.513 INFO:tasks.workunit.client.0.trial125.stderr:8 2026-02-25T22:23:23.513 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:23:23.513 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:23:23.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:23:23.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:23.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:23:23.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-25T22:23:23.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-25T22:23:23.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491' 2026-02-25T22:23:23.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:23.593 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:23:23.672 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672970 2026-02-25T22:23:23.672 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672970 2026-02-25T22:23:23.672 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2026-02-25T22:23:23.672 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:23.672 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:23:23.751 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509449 2026-02-25T22:23:23.751 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509449 2026-02-25T22:23:23.751 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449' 2026-02-25T22:23:23.751 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:23.752 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:23:23.831 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378633 2026-02-25T22:23:23.831 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378633 2026-02-25T22:23:23.831 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633' 2026-02-25T22:23:23.831 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:23.831 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:23:23.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247816 2026-02-25T22:23:23.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247816 2026-02-25T22:23:23.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816' 2026-02-25T22:23:23.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:23.910 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:23:23.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117000 2026-02-25T22:23:23.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117000 2026-02-25T22:23:23.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816 5-115964117000' 2026-02-25T22:23:23.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:23.988 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:23:24.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986182 2026-02-25T22:23:24.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986182 2026-02-25T22:23:24.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816 5-115964117000 6-133143986182' 2026-02-25T22:23:24.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:24.068 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T22:23:24.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822662 2026-02-25T22:23:24.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822662 2026-02-25T22:23:24.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816 5-115964117000 6-133143986182 7-154618822662' 2026-02-25T22:23:24.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:24.149 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T22:23:24.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691845 2026-02-25T22:23:24.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691845 2026-02-25T22:23:24.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816 5-115964117000 6-133143986182 7-154618822662 8-171798691845' 2026-02-25T22:23:24.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:24.228 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T22:23:24.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561029 2026-02-25T22:23:24.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561029 2026-02-25T22:23:24.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816 5-115964117000 6-133143986182 7-154618822662 8-171798691845 9-188978561029' 2026-02-25T22:23:24.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:24.306 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T22:23:24.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397507 2026-02-25T22:23:24.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397507 2026-02-25T22:23:24.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-81604378633 4-98784247816 5-115964117000 6-133143986182 7-154618822662 8-171798691845 9-188978561029 10-210453397507' 2026-02-25T22:23:24.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:24.386 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836491 2026-02-25T22:23:24.386 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:24.387 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:23:24.388 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-25T22:23:24.388 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:24.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836491 2026-02-25T22:23:24.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-25T22:23:24.390 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 21474836491 2026-02-25T22:23:24.390 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:23:24.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-25T22:23:24.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:23:25.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:23:25.612 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:23:25.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-25T22:23:25.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:25.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672970 2026-02-25T22:23:25.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:25.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:23:25.825 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672970 2026-02-25T22:23:25.825 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:25.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672970 2026-02-25T22:23:25.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-25T22:23:25.827 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 42949672970 2026-02-25T22:23:25.827 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:23:26.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-25T22:23:26.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:26.050 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-64424509449 2026-02-25T22:23:26.050 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:26.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:23:26.052 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509449 2026-02-25T22:23:26.052 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:26.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=64424509449 2026-02-25T22:23:26.054 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 64424509449 2026-02-25T22:23:26.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2026-02-25T22:23:26.054 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:23:26.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 64424509450 -lt 64424509449 2026-02-25T22:23:26.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:26.271 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-81604378633 2026-02-25T22:23:26.271 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:26.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:23:26.273 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378633 2026-02-25T22:23:26.273 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:26.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378633 2026-02-25T22:23:26.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 81604378633' 2026-02-25T22:23:26.275 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 81604378633 2026-02-25T22:23:26.276 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:23:26.491 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378633 -lt 81604378633 2026-02-25T22:23:26.491 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:26.492 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247816 2026-02-25T22:23:26.492 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:26.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:23:26.494 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247816 2026-02-25T22:23:26.494 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:26.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247816 2026-02-25T22:23:26.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247816' 2026-02-25T22:23:26.496 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 98784247816 2026-02-25T22:23:26.496 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:23:26.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247817 -lt 98784247816 2026-02-25T22:23:26.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:26.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-115964117000 2026-02-25T22:23:26.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:26.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:23:26.721 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117000 2026-02-25T22:23:26.721 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:26.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117000 2026-02-25T22:23:26.723 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 115964117000 2026-02-25T22:23:26.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 115964117000' 2026-02-25T22:23:26.723 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:23:26.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117001 -lt 115964117000 2026-02-25T22:23:26.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:26.937 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-133143986182 2026-02-25T22:23:26.937 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:26.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:23:26.939 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986182 2026-02-25T22:23:26.939 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:26.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=133143986182 2026-02-25T22:23:26.940 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 133143986182 2026-02-25T22:23:26.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 133143986182' 2026-02-25T22:23:26.941 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:23:27.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 133143986183 -lt 133143986182 2026-02-25T22:23:27.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:27.146 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-154618822662 2026-02-25T22:23:27.146 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:27.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T22:23:27.148 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822662 2026-02-25T22:23:27.148 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:27.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=154618822662 2026-02-25T22:23:27.150 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.7 seq 154618822662 2026-02-25T22:23:27.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 154618822662' 2026-02-25T22:23:27.150 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T22:23:27.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 154618822662 -lt 154618822662 2026-02-25T22:23:27.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:27.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-171798691845 2026-02-25T22:23:27.376 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:27.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T22:23:27.377 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691845 2026-02-25T22:23:27.378 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:27.379 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691845 2026-02-25T22:23:27.379 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.8 seq 171798691845 2026-02-25T22:23:27.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 171798691845' 2026-02-25T22:23:27.380 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T22:23:27.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691845 -lt 171798691845 2026-02-25T22:23:27.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:27.594 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-188978561029 2026-02-25T22:23:27.594 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:27.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T22:23:27.596 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-188978561029 2026-02-25T22:23:27.596 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:27.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561029 2026-02-25T22:23:27.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 188978561029' 2026-02-25T22:23:27.598 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.9 seq 188978561029 2026-02-25T22:23:27.598 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T22:23:27.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561030 -lt 188978561029 2026-02-25T22:23:27.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:27.815 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-210453397507 2026-02-25T22:23:27.815 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:27.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T22:23:27.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-210453397507 2026-02-25T22:23:27.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:27.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397507 2026-02-25T22:23:27.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 210453397507' 2026-02-25T22:23:27.819 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.10 seq 210453397507 2026-02-25T22:23:27.819 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T22:23:28.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397508 -lt 210453397507 2026-02-25T22:23:28.042 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:23:28.042 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:23:28.043 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:23:28.318 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:23:28.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:23:28.319 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:23:28.319 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:23:28.320 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:23:28.320 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:23:28.320 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:23:28.320 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:23:28.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:23:28.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:23:28.529 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:23:28.529 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:23:28.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:23:28.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:23:28.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:23:28.812 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2026-02-25T22:23:28.812 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:23:28.812 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:23:28.812 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:23:28.812 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:28.812 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:23:28.812 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.45907/ceph-osd.0.asok 2026-02-25T22:23:28.813 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2026-02-25T22:23:28.813 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.45907/ceph-osd.0.asok log flush 2026-02-25T22:23:28.869 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:23:28.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2026-02-25T22:23:28.876 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:22:39.922+0000 7fcf0cdb1a40 0 load: jerasure load: lrc 2026-02-25T22:23:28.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2026-02-25T22:23:28.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2026-02-25T22:23:28.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2026-02-25T22:23:29.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2026-02-25T22:23:29.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2026-02-25T22:23:29.500 INFO:tasks.workunit.client.0.trial125.stderr:pool 'ecpool' already exists 2026-02-25T22:23:29.510 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:23:30.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:23:30.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:23:30.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:23:30.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:23:30.512 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:23:30.513 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:23:30.513 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:23:30.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:23:30.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:23:30.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:23:30.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:23:30.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:23:30.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:23:30.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:23:30.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:23:30.611 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:23:30.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:23:30.818 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:23:30.818 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:23:30.818 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:23:30.818 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:23:30.818 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:23:30.818 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:23:30.819 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:23:30.819 INFO:tasks.workunit.client.0.trial125.stderr:8 2026-02-25T22:23:30.819 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:23:30.819 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:23:30.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:23:30.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:30.819 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:23:30.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836494 2026-02-25T22:23:30.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836494 2026-02-25T22:23:30.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836494' 2026-02-25T22:23:30.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:30.898 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:23:30.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672974 2026-02-25T22:23:30.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672974 2026-02-25T22:23:30.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836494 1-42949672974' 2026-02-25T22:23:30.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:30.975 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:23:31.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509453 2026-02-25T22:23:31.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509453 2026-02-25T22:23:31.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-64424509453' 2026-02-25T22:23:31.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:31.053 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:23:31.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378636 2026-02-25T22:23:31.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378636 2026-02-25T22:23:31.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-64424509453 3-81604378636' 2026-02-25T22:23:31.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:31.130 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:23:31.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247819 2026-02-25T22:23:31.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247819 2026-02-25T22:23:31.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-64424509453 3-81604378636 4-98784247819' 2026-02-25T22:23:31.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:31.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:23:31.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117004 2026-02-25T22:23:31.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117004 2026-02-25T22:23:31.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-64424509453 3-81604378636 4-98784247819 5-115964117004' 2026-02-25T22:23:31.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:31.289 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:23:31.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986186 2026-02-25T22:23:31.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986186 2026-02-25T22:23:31.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-64424509453 3-81604378636 4-98784247819 5-115964117004 6-133143986186' 2026-02-25T22:23:31.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:31.365 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T22:23:31.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822665 2026-02-25T22:23:31.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822665 2026-02-25T22:23:31.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-64424509453 3-81604378636 4-98784247819 5-115964117004 6-133143986186 7-154618822665' 2026-02-25T22:23:31.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:31.445 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T22:23:31.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691848 2026-02-25T22:23:31.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691848 2026-02-25T22:23:31.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-64424509453 3-81604378636 4-98784247819 5-115964117004 6-133143986186 7-154618822665 8-171798691848' 2026-02-25T22:23:31.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:31.521 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T22:23:31.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561033 2026-02-25T22:23:31.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561033 2026-02-25T22:23:31.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-64424509453 3-81604378636 4-98784247819 5-115964117004 6-133143986186 7-154618822665 8-171798691848 9-188978561033' 2026-02-25T22:23:31.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:31.601 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T22:23:31.677 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397511 2026-02-25T22:23:31.677 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397511 2026-02-25T22:23:31.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-64424509453 3-81604378636 4-98784247819 5-115964117004 6-133143986186 7-154618822665 8-171798691848 9-188978561033 10-210453397511' 2026-02-25T22:23:31.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:31.678 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836494 2026-02-25T22:23:31.678 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:31.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:23:31.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836494 2026-02-25T22:23:31.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:31.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836494 2026-02-25T22:23:31.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2026-02-25T22:23:31.682 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 21474836494 2026-02-25T22:23:31.682 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:23:31.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836492 -lt 21474836494 2026-02-25T22:23:31.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:23:32.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:23:32.897 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:23:33.106 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836495 -lt 21474836494 2026-02-25T22:23:33.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:33.107 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672974 2026-02-25T22:23:33.108 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:33.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:23:33.110 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672974 2026-02-25T22:23:33.110 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:33.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672974 2026-02-25T22:23:33.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2026-02-25T22:23:33.112 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 42949672974 2026-02-25T22:23:33.112 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:23:33.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672974 -lt 42949672974 2026-02-25T22:23:33.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:33.330 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-64424509453 2026-02-25T22:23:33.330 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:33.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:23:33.332 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509453 2026-02-25T22:23:33.332 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:33.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=64424509453 2026-02-25T22:23:33.333 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 64424509453 2026-02-25T22:23:33.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2026-02-25T22:23:33.334 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:23:33.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 64424509453 -lt 64424509453 2026-02-25T22:23:33.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:33.552 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-81604378636 2026-02-25T22:23:33.552 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:33.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:23:33.554 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378636 2026-02-25T22:23:33.554 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:33.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378636 2026-02-25T22:23:33.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 81604378636' 2026-02-25T22:23:33.556 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 81604378636 2026-02-25T22:23:33.556 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:23:33.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378637 -lt 81604378636 2026-02-25T22:23:33.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:33.778 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247819 2026-02-25T22:23:33.778 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:33.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:23:33.780 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247819 2026-02-25T22:23:33.780 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:33.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247819 2026-02-25T22:23:33.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247819' 2026-02-25T22:23:33.782 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 98784247819 2026-02-25T22:23:33.782 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:23:33.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247820 -lt 98784247819 2026-02-25T22:23:33.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:33.999 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-115964117004 2026-02-25T22:23:33.999 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:34.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:23:34.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117004 2026-02-25T22:23:34.002 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:34.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117004 2026-02-25T22:23:34.003 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 115964117004 2026-02-25T22:23:34.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 115964117004' 2026-02-25T22:23:34.003 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:23:34.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117004 -lt 115964117004 2026-02-25T22:23:34.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:34.217 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-133143986186 2026-02-25T22:23:34.217 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:34.218 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:23:34.219 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986186 2026-02-25T22:23:34.219 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:34.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=133143986186 2026-02-25T22:23:34.221 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 133143986186 2026-02-25T22:23:34.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 133143986186' 2026-02-25T22:23:34.221 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:23:34.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 133143986186 -lt 133143986186 2026-02-25T22:23:34.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:34.435 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-154618822665 2026-02-25T22:23:34.435 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:34.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T22:23:34.437 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822665 2026-02-25T22:23:34.437 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:34.439 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=154618822665 2026-02-25T22:23:34.439 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 154618822665' 2026-02-25T22:23:34.439 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.7 seq 154618822665 2026-02-25T22:23:34.439 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T22:23:34.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 154618822665 -lt 154618822665 2026-02-25T22:23:34.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:34.663 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-171798691848 2026-02-25T22:23:34.663 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:34.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T22:23:34.665 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691848 2026-02-25T22:23:34.666 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:34.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691848 2026-02-25T22:23:34.667 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.8 seq 171798691848 2026-02-25T22:23:34.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 171798691848' 2026-02-25T22:23:34.668 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T22:23:34.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691849 -lt 171798691848 2026-02-25T22:23:34.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:34.886 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-188978561033 2026-02-25T22:23:34.886 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:34.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T22:23:34.888 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-188978561033 2026-02-25T22:23:34.888 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:34.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561033 2026-02-25T22:23:34.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 188978561033' 2026-02-25T22:23:34.890 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.9 seq 188978561033 2026-02-25T22:23:34.890 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T22:23:35.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561033 -lt 188978561033 2026-02-25T22:23:35.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:35.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-210453397511 2026-02-25T22:23:35.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:35.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T22:23:35.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-210453397511 2026-02-25T22:23:35.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:35.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397511 2026-02-25T22:23:35.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 210453397511' 2026-02-25T22:23:35.121 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.10 seq 210453397511 2026-02-25T22:23:35.121 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T22:23:35.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397511 -lt 210453397511 2026-02-25T22:23:35.332 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:23:35.332 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:23:35.332 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:23:35.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 16 == 0 2026-02-25T22:23:35.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:23:35.608 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:23:35.608 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:23:35.608 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:23:35.608 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:23:35.608 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:23:35.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:23:35.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=16 2026-02-25T22:23:35.815 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:23:35.815 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:23:35.815 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:23:36.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 16 = 16 2026-02-25T22:23:36.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:23:36.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:23:36.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2026-02-25T22:23:36.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T22:23:36.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2026-02-25T22:23:36.099 INFO:tasks.workunit.client.0.trial125.stderr:TEST_chunk_mapping 2026-02-25T22:23:36.100 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_put_get_isa 2026-02-25T22:23:36.100 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_put_get_jerasure 2026-02-25T22:23:36.100 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_put_get_lrc_advanced 2026-02-25T22:23:36.100 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_put_get_lrc_kml 2026-02-25T22:23:36.100 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_put_get_shec' 2026-02-25T22:23:36.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T22:23:36.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2026-02-25T22:23:36.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: TEST_alignment_constraints: local payload=ABC 2026-02-25T22:23:36.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:267: TEST_alignment_constraints: echo ABC 2026-02-25T22:23:36.101 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:273: TEST_alignment_constraints: chunk_size 2026-02-25T22:23:36.101 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-25T22:23:36.110 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: local chunk_size=0 2026-02-25T22:23:36.110 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: chunk_size: '[' 0 -eq 0 ']' 2026-02-25T22:23:36.110 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: ceph-conf --show-config-value osd_pool_default_flag_ec_optimizations 2026-02-25T22:23:36.121 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: local ec_opt=false 2026-02-25T22:23:36.121 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: chunk_size: '[' false = true ']' 2026-02-25T22:23:36.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: chunk_size: chunk_size=4096 2026-02-25T22:23:36.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: chunk_size: echo 4096 2026-02-25T22:23:36.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:273: TEST_alignment_constraints: local stripe_unit=4096 2026-02-25T22:23:36.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:274: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2026-02-25T22:23:36.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:274: TEST_alignment_constraints: grep k= 2026-02-25T22:23:36.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:274: TEST_alignment_constraints: eval local k=2 2026-02-25T22:23:36.342 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:274: TEST_alignment_constraints: local k=2 2026-02-25T22:23:36.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:275: TEST_alignment_constraints: local block_size=8191 2026-02-25T22:23:36.343 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:276: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2026-02-25T22:23:36.343 INFO:tasks.workunit.client.0.trial125.stderr:2+0 records in 2026-02-25T22:23:36.343 INFO:tasks.workunit.client.0.trial125.stderr:2+0 records out 2026-02-25T22:23:36.344 INFO:tasks.workunit.client.0.trial125.stderr:16382 bytes (16 kB, 16 KiB) copied, 9.9723e-05 s, 164 MB/s 2026-02-25T22:23:36.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:277: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2026-02-25T22:23:36.366 INFO:tasks.workunit.client.0.trial125.stderr:INFO: op_size has been rounded to 8192 2026-02-25T22:23:36.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:279: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2026-02-25T22:23:36.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T22:23:36.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2026-02-25T22:23:36.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:313: TEST_chunk_mapping: local dir=td/test-erasure-code 2026-02-25T22:23:36.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:320: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2026-02-25T22:23:36.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: local dir=td/test-erasure-code 2026-02-25T22:23:36.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local poolname=ecpool 2026-02-25T22:23:36.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: local first=0 2026-02-25T22:23:36.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: local second=1 2026-02-25T22:23:36.374 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: chunk_size 2026-02-25T22:23:36.375 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-25T22:23:36.384 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: local chunk_size=0 2026-02-25T22:23:36.384 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: chunk_size: '[' 0 -eq 0 ']' 2026-02-25T22:23:36.384 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: ceph-conf --show-config-value osd_pool_default_flag_ec_optimizations 2026-02-25T22:23:36.395 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: local ec_opt=false 2026-02-25T22:23:36.395 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: chunk_size: '[' false = true ']' 2026-02-25T22:23:36.395 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: chunk_size: chunk_size=4096 2026-02-25T22:23:36.395 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: chunk_size: echo 4096 2026-02-25T22:23:36.395 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2026-02-25T22:23:36.396 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: chunk_size 2026-02-25T22:23:36.396 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-25T22:23:36.404 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: local chunk_size=0 2026-02-25T22:23:36.404 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: chunk_size: '[' 0 -eq 0 ']' 2026-02-25T22:23:36.405 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: ceph-conf --show-config-value osd_pool_default_flag_ec_optimizations 2026-02-25T22:23:36.417 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: local ec_opt=false 2026-02-25T22:23:36.417 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: chunk_size: '[' false = true ']' 2026-02-25T22:23:36.418 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: chunk_size: chunk_size=4096 2026-02-25T22:23:36.418 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: chunk_size: echo 4096 2026-02-25T22:23:36.418 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2026-02-25T22:23:36.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2026-02-25T22:23:36.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2026-02-25T22:23:36.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2026-02-25T22:23:36.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2026-02-25T22:23:36.466 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2026-02-25T22:23:36.466 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-25T22:23:36.466 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGecpool 2026-02-25T22:23:36.467 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2026-02-25T22:23:36.467 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:23:36.673 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:23:36.673 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:23:36.673 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:23:36.673 INFO:tasks.workunit.client.0.trial125.stderr:0' 2026-02-25T22:23:36.673 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 2026-02-25T22:23:36.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: osds=('3' '5' '9' '0') 2026-02-25T22:23:36.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: local -a osds 2026-02-25T22:23:36.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i = 0 )) 2026-02-25T22:23:36.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 4 )) 2026-02-25T22:23:36.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2026-02-25T22:23:36.737 INFO:tasks.workunit.client.0.trial125.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T22:23:36.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T22:23:36.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 4 )) 2026-02-25T22:23:36.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2026-02-25T22:23:36.811 INFO:tasks.workunit.client.0.trial125.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T22:23:36.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T22:23:36.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 4 )) 2026-02-25T22:23:36.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2026-02-25T22:23:36.878 INFO:tasks.workunit.client.0.trial125.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T22:23:36.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T22:23:36.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 4 )) 2026-02-25T22:23:36.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2026-02-25T22:23:36.954 INFO:tasks.workunit.client.0.trial125.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T22:23:36.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T22:23:36.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 4 )) 2026-02-25T22:23:36.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:23:36.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:305: verify_chunk_mapping: rm td/test-erasure-code/COPY 2026-02-25T22:23:36.959 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2026-02-25T22:23:36.959 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-25T22:23:36.959 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGecpool 2026-02-25T22:23:36.960 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2026-02-25T22:23:36.960 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:23:37.177 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:23:37.177 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:23:37.177 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:23:37.177 INFO:tasks.workunit.client.0.trial125.stderr:0' 2026-02-25T22:23:37.177 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 2026-02-25T22:23:37.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: osds=('3' '5' '9' '0') 2026-02-25T22:23:37.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: local -a osds 2026-02-25T22:23:37.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:308: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2026-02-25T22:23:37.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2026-02-25T22:23:37.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:23:37.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-25T22:23:37.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:23:37.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2026-02-25T22:23:37.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-25T22:23:37.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:23:37.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-25T22:23:37.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:23:37.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2026-02-25T22:23:37.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:308: verify_chunk_mapping: grep --quiet FIRSTecpool 2026-02-25T22:23:37.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:23:37.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:23:37.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:23:37.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:23:37.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:23:37.484 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:23:37.484 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2026-02-25T22:23:37.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-25T22:23:37.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:23:37.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-25T22:23:37.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:23:37.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2026-02-25T22:23:37.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2026-02-25T22:23:37.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2026-02-25T22:23:37.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2026-02-25T22:23:37.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:23:37.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T22:23:37.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:23:37.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/3 2026-02-25T22:23:37.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:23:37.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:23:37.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:23:37.704 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:37.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:37.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:23:37.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:23:37.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:23:37.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:23:37.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:23:37.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:23:37.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:23:37.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:23:37.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:23:37.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/3 2026-02-25T22:23:37.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T22:23:37.706 INFO:tasks.workunit.client.0.trial125.stderr:start osd.3 2026-02-25T22:23:37.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:23:37.707 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/3/whoami 2026-02-25T22:23:37.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T22:23:37.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:23:37.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:23:37.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:23:37.724 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:37.724+0000 7fe831e70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:37.726 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:37.724+0000 7fe831e70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:37.727 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:37.724+0000 7fe831e70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:37.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T22:23:37.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:23:37.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:23:37.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:23:37.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:23:37.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:37.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:23:37.917 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:23:37.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:37.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:23:38.127 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:38.134 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:38.132+0000 7fe831e70a40 -1 Falling back to public interface 2026-02-25T22:23:38.763 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:38.760+0000 7fe831e70a40 -1 osd.3 66 log_to_monitors true 2026-02-25T22:23:39.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:39.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:39.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:23:39.129 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:23:39.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:39.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:23:39.338 INFO:tasks.workunit.client.0.trial125.stderr:osd.3 up in weight 1 up_from 69 up_thru 69 down_at 66 last_clean_interval [19,66) [v2:127.0.0.1:6826/973639295,v1:127.0.0.1:6827/973639295] [v2:127.0.0.1:6828/973639295,v1:127.0.0.1:6829/973639295] exists,up c248833a-f8a8-4136-a10f-63738620755b 2026-02-25T22:23:39.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:23:39.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:23:39.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:23:39.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:23:39.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:23:39.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:23:39.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:23:39.339 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:23:39.340 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:23:39.340 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:23:39.340 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:23:39.340 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:23:39.341 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:23:39.440 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:23:39.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:23:39.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:23:39.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:23:39.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:23:39.441 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:23:39.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:23:39.651 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:23:39.651 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:23:39.651 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:23:39.651 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:23:39.651 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:23:39.651 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:23:39.651 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:23:39.651 INFO:tasks.workunit.client.0.trial125.stderr:8 2026-02-25T22:23:39.651 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:23:39.652 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:23:39.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:23:39.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:39.652 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:23:39.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836498 2026-02-25T22:23:39.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836498 2026-02-25T22:23:39.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836498' 2026-02-25T22:23:39.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:39.731 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:23:39.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672977 2026-02-25T22:23:39.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672977 2026-02-25T22:23:39.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836498 1-42949672977' 2026-02-25T22:23:39.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:39.810 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:23:39.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509457 2026-02-25T22:23:39.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509457 2026-02-25T22:23:39.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457' 2026-02-25T22:23:39.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:39.886 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:23:39.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743427 2026-02-25T22:23:39.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743427 2026-02-25T22:23:39.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457 3-296352743427' 2026-02-25T22:23:39.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:39.963 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:23:40.044 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247823 2026-02-25T22:23:40.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247823 2026-02-25T22:23:40.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457 3-296352743427 4-98784247823' 2026-02-25T22:23:40.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:40.045 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:23:40.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117007 2026-02-25T22:23:40.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117007 2026-02-25T22:23:40.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457 3-296352743427 4-98784247823 5-115964117007' 2026-02-25T22:23:40.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:40.123 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:23:40.200 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986190 2026-02-25T22:23:40.200 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986190 2026-02-25T22:23:40.200 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457 3-296352743427 4-98784247823 5-115964117007 6-133143986190' 2026-02-25T22:23:40.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:40.201 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T22:23:40.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822669 2026-02-25T22:23:40.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822669 2026-02-25T22:23:40.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457 3-296352743427 4-98784247823 5-115964117007 6-133143986190 7-154618822669' 2026-02-25T22:23:40.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:40.280 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T22:23:40.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691852 2026-02-25T22:23:40.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691852 2026-02-25T22:23:40.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457 3-296352743427 4-98784247823 5-115964117007 6-133143986190 7-154618822669 8-171798691852' 2026-02-25T22:23:40.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:40.358 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T22:23:40.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561036 2026-02-25T22:23:40.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561036 2026-02-25T22:23:40.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457 3-296352743427 4-98784247823 5-115964117007 6-133143986190 7-154618822669 8-171798691852 9-188978561036' 2026-02-25T22:23:40.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:40.437 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T22:23:40.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397514 2026-02-25T22:23:40.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397514 2026-02-25T22:23:40.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509457 3-296352743427 4-98784247823 5-115964117007 6-133143986190 7-154618822669 8-171798691852 9-188978561036 10-210453397514' 2026-02-25T22:23:40.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:40.517 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836498 2026-02-25T22:23:40.517 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:40.517 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:23:40.518 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836498 2026-02-25T22:23:40.518 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:40.519 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836498 2026-02-25T22:23:40.519 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2026-02-25T22:23:40.519 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 21474836498 2026-02-25T22:23:40.520 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:23:40.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836498 -lt 21474836498 2026-02-25T22:23:40.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:40.743 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672977 2026-02-25T22:23:40.743 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:40.744 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:23:40.744 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672977 2026-02-25T22:23:40.744 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:40.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672977 2026-02-25T22:23:40.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2026-02-25T22:23:40.746 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 42949672977 2026-02-25T22:23:40.746 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:23:40.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672978 -lt 42949672977 2026-02-25T22:23:40.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:40.955 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-64424509457 2026-02-25T22:23:40.956 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:40.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:23:40.957 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509457 2026-02-25T22:23:40.958 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:40.959 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=64424509457 2026-02-25T22:23:40.959 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2026-02-25T22:23:40.959 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 64424509457 2026-02-25T22:23:40.959 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:23:41.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 64424509457 -lt 64424509457 2026-02-25T22:23:41.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:41.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-296352743427 2026-02-25T22:23:41.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:41.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:23:41.164 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-296352743427 2026-02-25T22:23:41.164 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:41.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=296352743427 2026-02-25T22:23:41.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 296352743427' 2026-02-25T22:23:41.166 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 296352743427 2026-02-25T22:23:41.167 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:23:41.382 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 296352743427 -lt 296352743427 2026-02-25T22:23:41.382 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:41.382 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247823 2026-02-25T22:23:41.382 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:41.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:23:41.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247823 2026-02-25T22:23:41.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:41.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247823 2026-02-25T22:23:41.386 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247823' 2026-02-25T22:23:41.386 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.4 seq 98784247823 2026-02-25T22:23:41.386 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:23:41.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247823 -lt 98784247823 2026-02-25T22:23:41.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:41.601 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-115964117007 2026-02-25T22:23:41.601 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:41.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:23:41.603 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117007 2026-02-25T22:23:41.603 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:41.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117007 2026-02-25T22:23:41.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 115964117007' 2026-02-25T22:23:41.604 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.5 seq 115964117007 2026-02-25T22:23:41.605 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:23:41.813 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117008 -lt 115964117007 2026-02-25T22:23:41.813 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:41.814 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-133143986190 2026-02-25T22:23:41.814 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:41.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:23:41.816 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986190 2026-02-25T22:23:41.816 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:41.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=133143986190 2026-02-25T22:23:41.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 133143986190' 2026-02-25T22:23:41.818 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.6 seq 133143986190 2026-02-25T22:23:41.818 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:23:42.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 133143986190 -lt 133143986190 2026-02-25T22:23:42.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:42.033 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-154618822669 2026-02-25T22:23:42.033 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:42.034 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T22:23:42.035 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822669 2026-02-25T22:23:42.035 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:42.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=154618822669 2026-02-25T22:23:42.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 154618822669' 2026-02-25T22:23:42.036 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.7 seq 154618822669 2026-02-25T22:23:42.037 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T22:23:42.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 154618822669 -lt 154618822669 2026-02-25T22:23:42.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:42.253 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-171798691852 2026-02-25T22:23:42.253 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:42.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T22:23:42.255 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691852 2026-02-25T22:23:42.255 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:42.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691852 2026-02-25T22:23:42.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 171798691852' 2026-02-25T22:23:42.256 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.8 seq 171798691852 2026-02-25T22:23:42.257 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T22:23:42.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691852 -lt 171798691852 2026-02-25T22:23:42.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:42.474 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-188978561036 2026-02-25T22:23:42.474 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:42.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T22:23:42.476 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-188978561036 2026-02-25T22:23:42.476 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:42.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561036 2026-02-25T22:23:42.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 188978561036' 2026-02-25T22:23:42.478 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.9 seq 188978561036 2026-02-25T22:23:42.478 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T22:23:42.694 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561036 -lt 188978561036 2026-02-25T22:23:42.694 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:42.695 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-210453397514 2026-02-25T22:23:42.695 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:42.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T22:23:42.696 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-210453397514 2026-02-25T22:23:42.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:42.698 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397514 2026-02-25T22:23:42.698 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 210453397514' 2026-02-25T22:23:42.698 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.10 seq 210453397514 2026-02-25T22:23:42.698 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T22:23:42.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397515 -lt 210453397514 2026-02-25T22:23:42.921 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:23:42.921 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:23:42.921 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:23:43.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 16 == 0 2026-02-25T22:23:43.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:23:43.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:23:43.201 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:23:43.201 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:23:43.201 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:23:43.201 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:23:43.201 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:23:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=15 2026-02-25T22:23:43.415 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:23:43.416 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:23:43.416 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:23:43.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 15 = 16 2026-02-25T22:23:43.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 15 '!=' -1 2026-02-25T22:23:43.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-25T22:23:43.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=15 2026-02-25T22:23:43.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:23:43.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-25T22:23:43.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:23:43.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:23:43.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:23:43.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:23:43.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:23:43.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:23:43.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:23:43.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:23:44.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=15 2026-02-25T22:23:44.007 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:23:44.008 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:23:44.008 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:23:44.292 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 15 = 16 2026-02-25T22:23:44.293 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 15 '!=' 15 2026-02-25T22:23:44.293 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:23:44.293 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:23:44.293 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:23:44.293 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:23:44.293 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:23:44.293 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:23:44.293 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:23:44.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:23:44.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:23:44.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-25T22:23:44.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:23:44.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-25T22:23:44.785 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:23:44.785 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:23:44.785 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:23:44.785 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:23:44.785 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:23:44.785 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:23:44.786 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:23:44.786 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:23:44.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=16 2026-02-25T22:23:44.993 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:23:44.994 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:23:44.994 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:23:45.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 16 = 16 2026-02-25T22:23:45.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:23:45.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:23:45.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:309: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2026-02-25T22:23:45.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:309: verify_chunk_mapping: grep --quiet SECONDecpool 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:23:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:23:45.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:23:45.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:23:45.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:23:45.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2026-02-25T22:23:45.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-25T22:23:45.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:23:45.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-25T22:23:45.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:23:45.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2026-02-25T22:23:45.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2026-02-25T22:23:45.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2026-02-25T22:23:45.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2026-02-25T22:23:45.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:23:45.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-25T22:23:45.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:23:45.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/5 2026-02-25T22:23:45.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:23:45.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:23:45.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:23:45.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:45.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/5 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-25T22:23:45.762 INFO:tasks.workunit.client.0.trial125.stderr:start osd.5 2026-02-25T22:23:45.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:23:45.764 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/5/whoami 2026-02-25T22:23:45.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-25T22:23:45.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:23:45.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:23:45.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:23:45.778 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:45.775+0000 7f958b8e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:45.780 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:45.775+0000 7f958b8e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:45.781 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:45.779+0000 7f958b8e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:45.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-25T22:23:45.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:23:45.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T22:23:45.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:23:45.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:23:45.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:45.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:23:45.969 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:23:45.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:45.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:23:46.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:46.186 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:46.183+0000 7f958b8e0a40 -1 Falling back to public interface 2026-02-25T22:23:46.849 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:46.847+0000 7f958b8e0a40 -1 osd.5 72 log_to_monitors true 2026-02-25T22:23:47.183 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:47.183 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:47.183 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:23:47.183 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:23:47.183 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:47.183 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:23:47.397 INFO:tasks.workunit.client.0.trial125.stderr:osd.5 up in weight 1 up_from 76 up_thru 76 down_at 73 last_clean_interval [27,72) [v2:127.0.0.1:6842/3017870963,v1:127.0.0.1:6843/3017870963] [v2:127.0.0.1:6844/3017870963,v1:127.0.0.1:6845/3017870963] exists,up c1450571-a937-48d3-b9df-6988c9294fc6 2026-02-25T22:23:47.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:23:47.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:23:47.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:23:47.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:23:47.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:23:47.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:23:47.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:23:47.397 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:23:47.398 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:23:47.398 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:23:47.398 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:23:47.399 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:23:47.399 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:23:47.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:23:47.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:23:47.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:23:47.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:23:47.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:23:47.499 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:23:47.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:23:47.703 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:23:47.703 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:23:47.703 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:23:47.703 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:23:47.703 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:23:47.703 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:23:47.703 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:23:47.703 INFO:tasks.workunit.client.0.trial125.stderr:8 2026-02-25T22:23:47.704 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:23:47.704 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:23:47.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:23:47.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:47.704 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:23:47.782 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836502 2026-02-25T22:23:47.782 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836502 2026-02-25T22:23:47.782 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836502' 2026-02-25T22:23:47.783 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:47.783 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:23:47.860 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672981 2026-02-25T22:23:47.860 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672981 2026-02-25T22:23:47.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836502 1-42949672981' 2026-02-25T22:23:47.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:47.861 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:23:47.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509460 2026-02-25T22:23:47.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509460 2026-02-25T22:23:47.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509460' 2026-02-25T22:23:47.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:47.939 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:23:48.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743430 2026-02-25T22:23:48.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743430 2026-02-25T22:23:48.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509460 3-296352743430' 2026-02-25T22:23:48.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:48.017 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:23:48.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247827 2026-02-25T22:23:48.097 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247827 2026-02-25T22:23:48.097 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509460 3-296352743430 4-98784247827' 2026-02-25T22:23:48.097 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:48.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:23:48.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514499 2026-02-25T22:23:48.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514499 2026-02-25T22:23:48.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509460 3-296352743430 4-98784247827 5-326417514499' 2026-02-25T22:23:48.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:48.177 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:23:48.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986193 2026-02-25T22:23:48.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986193 2026-02-25T22:23:48.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509460 3-296352743430 4-98784247827 5-326417514499 6-133143986193' 2026-02-25T22:23:48.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:48.253 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T22:23:48.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822672 2026-02-25T22:23:48.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822672 2026-02-25T22:23:48.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509460 3-296352743430 4-98784247827 5-326417514499 6-133143986193 7-154618822672' 2026-02-25T22:23:48.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:48.333 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T22:23:48.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691856 2026-02-25T22:23:48.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691856 2026-02-25T22:23:48.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509460 3-296352743430 4-98784247827 5-326417514499 6-133143986193 7-154618822672 8-171798691856' 2026-02-25T22:23:48.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:48.410 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T22:23:48.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561040 2026-02-25T22:23:48.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561040 2026-02-25T22:23:48.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509460 3-296352743430 4-98784247827 5-326417514499 6-133143986193 7-154618822672 8-171798691856 9-188978561040' 2026-02-25T22:23:48.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:48.489 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T22:23:48.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397518 2026-02-25T22:23:48.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397518 2026-02-25T22:23:48.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509460 3-296352743430 4-98784247827 5-326417514499 6-133143986193 7-154618822672 8-171798691856 9-188978561040 10-210453397518' 2026-02-25T22:23:48.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:48.565 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836502 2026-02-25T22:23:48.565 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:48.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:23:48.567 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836502 2026-02-25T22:23:48.567 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:48.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836502 2026-02-25T22:23:48.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2026-02-25T22:23:48.568 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 21474836502 2026-02-25T22:23:48.569 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:23:48.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836502 -lt 21474836502 2026-02-25T22:23:48.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:48.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672981 2026-02-25T22:23:48.794 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:48.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:23:48.795 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672981 2026-02-25T22:23:48.795 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:48.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672981 2026-02-25T22:23:48.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2026-02-25T22:23:48.797 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 42949672981 2026-02-25T22:23:48.798 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:23:49.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672981 -lt 42949672981 2026-02-25T22:23:49.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:49.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-64424509460 2026-02-25T22:23:49.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:49.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:23:49.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509460 2026-02-25T22:23:49.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=64424509460 2026-02-25T22:23:49.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2026-02-25T22:23:49.013 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 64424509460 2026-02-25T22:23:49.014 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:23:49.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 64424509460 -lt 64424509460 2026-02-25T22:23:49.229 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:49.229 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-296352743430 2026-02-25T22:23:49.229 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:49.230 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:23:49.231 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-296352743430 2026-02-25T22:23:49.231 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:49.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=296352743430 2026-02-25T22:23:49.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 296352743430' 2026-02-25T22:23:49.233 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 296352743430 2026-02-25T22:23:49.233 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:23:49.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 296352743430 -lt 296352743430 2026-02-25T22:23:49.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:49.444 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247827 2026-02-25T22:23:49.445 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:49.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:23:49.446 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247827 2026-02-25T22:23:49.446 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:49.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247827 2026-02-25T22:23:49.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247827' 2026-02-25T22:23:49.448 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.4 seq 98784247827 2026-02-25T22:23:49.448 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:23:49.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247827 -lt 98784247827 2026-02-25T22:23:49.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:49.681 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-326417514499 2026-02-25T22:23:49.681 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:49.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:23:49.683 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-326417514499 2026-02-25T22:23:49.683 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:49.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=326417514499 2026-02-25T22:23:49.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 326417514499' 2026-02-25T22:23:49.685 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.5 seq 326417514499 2026-02-25T22:23:49.685 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:23:49.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 326417514499 -lt 326417514499 2026-02-25T22:23:49.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:49.899 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-133143986193 2026-02-25T22:23:49.899 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:49.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:23:49.901 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986193 2026-02-25T22:23:49.901 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:49.902 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=133143986193 2026-02-25T22:23:49.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 133143986193' 2026-02-25T22:23:49.903 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.6 seq 133143986193 2026-02-25T22:23:49.903 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:23:50.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 133143986193 -lt 133143986193 2026-02-25T22:23:50.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:50.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-154618822672 2026-02-25T22:23:50.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:50.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T22:23:50.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822672 2026-02-25T22:23:50.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:50.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=154618822672 2026-02-25T22:23:50.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 154618822672' 2026-02-25T22:23:50.123 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.7 seq 154618822672 2026-02-25T22:23:50.124 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T22:23:50.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 154618822673 -lt 154618822672 2026-02-25T22:23:50.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:50.332 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-171798691856 2026-02-25T22:23:50.332 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:50.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T22:23:50.334 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691856 2026-02-25T22:23:50.334 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:50.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691856 2026-02-25T22:23:50.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 171798691856' 2026-02-25T22:23:50.335 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.8 seq 171798691856 2026-02-25T22:23:50.335 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T22:23:50.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691856 -lt 171798691856 2026-02-25T22:23:50.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:50.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-188978561040 2026-02-25T22:23:50.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:50.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T22:23:50.549 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-188978561040 2026-02-25T22:23:50.549 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:50.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561040 2026-02-25T22:23:50.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 188978561040' 2026-02-25T22:23:50.551 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.9 seq 188978561040 2026-02-25T22:23:50.551 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T22:23:50.755 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561040 -lt 188978561040 2026-02-25T22:23:50.755 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:50.756 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-210453397518 2026-02-25T22:23:50.756 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:50.757 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T22:23:50.758 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-210453397518 2026-02-25T22:23:50.758 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:50.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397518 2026-02-25T22:23:50.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 210453397518' 2026-02-25T22:23:50.759 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.10 seq 210453397518 2026-02-25T22:23:50.760 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T22:23:50.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397518 -lt 210453397518 2026-02-25T22:23:50.976 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:23:50.976 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:23:50.976 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:23:51.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 16 == 0 2026-02-25T22:23:51.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:23:51.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:23:51.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:23:51.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:23:51.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:23:51.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:23:51.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:23:51.454 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=16 2026-02-25T22:23:51.454 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:23:51.455 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:23:51.455 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:23:51.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 16 = 16 2026-02-25T22:23:51.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:23:51.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:23:51.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:322: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2026-02-25T22:23:52.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:327: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2026-02-25T22:23:52.222 INFO:tasks.workunit.client.0.trial125.stdout:crush-device-class= 2026-02-25T22:23:52.223 INFO:tasks.workunit.client.0.trial125.stdout:crush-failure-domain=host 2026-02-25T22:23:52.223 INFO:tasks.workunit.client.0.trial125.stdout:crush-num-failure-domains=0 2026-02-25T22:23:52.223 INFO:tasks.workunit.client.0.trial125.stdout:crush-osds-per-failure-domain=0 2026-02-25T22:23:52.223 INFO:tasks.workunit.client.0.trial125.stdout:crush-root=default 2026-02-25T22:23:52.223 INFO:tasks.workunit.client.0.trial125.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2026-02-25T22:23:52.223 INFO:tasks.workunit.client.0.trial125.stdout:k=-1 2026-02-25T22:23:52.223 INFO:tasks.workunit.client.0.trial125.stdout:l=-1 2026-02-25T22:23:52.223 INFO:tasks.workunit.client.0.trial125.stdout:layers=[ [ "cDD", "" ] ] 2026-02-25T22:23:52.223 INFO:tasks.workunit.client.0.trial125.stdout:m=-1 2026-02-25T22:23:52.223 INFO:tasks.workunit.client.0.trial125.stdout:mapping=_DD 2026-02-25T22:23:52.223 INFO:tasks.workunit.client.0.trial125.stdout:plugin=lrc 2026-02-25T22:23:52.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:328: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2026-02-25T22:23:52.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2026-02-25T22:23:52.575 INFO:tasks.workunit.client.0.trial125.stderr:pool 'remap-pool' already exists 2026-02-25T22:23:52.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:23:53.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:336: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2026-02-25T22:23:53.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: local dir=td/test-erasure-code 2026-02-25T22:23:53.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local poolname=remap-pool 2026-02-25T22:23:53.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: local first=1 2026-02-25T22:23:53.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: local second=2 2026-02-25T22:23:53.588 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: chunk_size 2026-02-25T22:23:53.588 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-25T22:23:53.598 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: local chunk_size=0 2026-02-25T22:23:53.599 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: chunk_size: '[' 0 -eq 0 ']' 2026-02-25T22:23:53.599 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: ceph-conf --show-config-value osd_pool_default_flag_ec_optimizations 2026-02-25T22:23:53.614 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: local ec_opt=false 2026-02-25T22:23:53.614 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: chunk_size: '[' false = true ']' 2026-02-25T22:23:53.614 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: chunk_size: chunk_size=4096 2026-02-25T22:23:53.614 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: chunk_size: echo 4096 2026-02-25T22:23:53.614 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2026-02-25T22:23:53.615 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: chunk_size 2026-02-25T22:23:53.615 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-25T22:23:53.623 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: local chunk_size=0 2026-02-25T22:23:53.624 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: chunk_size: '[' 0 -eq 0 ']' 2026-02-25T22:23:53.624 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: ceph-conf --show-config-value osd_pool_default_flag_ec_optimizations 2026-02-25T22:23:53.634 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: local ec_opt=false 2026-02-25T22:23:53.634 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: chunk_size: '[' false = true ']' 2026-02-25T22:23:53.634 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: chunk_size: chunk_size=4096 2026-02-25T22:23:53.634 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: chunk_size: echo 4096 2026-02-25T22:23:53.634 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2026-02-25T22:23:53.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2026-02-25T22:23:53.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2026-02-25T22:23:53.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2026-02-25T22:23:53.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2026-02-25T22:23:53.678 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2026-02-25T22:23:53.678 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=remap-pool 2026-02-25T22:23:53.678 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGremap-pool 2026-02-25T22:23:53.679 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2026-02-25T22:23:53.679 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:23:53.886 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2026-02-25T22:23:53.886 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:23:53.887 INFO:tasks.workunit.client.0.trial125.stderr:9' 2026-02-25T22:23:53.887 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 6 9 2026-02-25T22:23:53.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: osds=('4' '6' '9') 2026-02-25T22:23:53.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: local -a osds 2026-02-25T22:23:53.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i = 0 )) 2026-02-25T22:23:53.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 3 )) 2026-02-25T22:23:53.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2026-02-25T22:23:53.955 INFO:tasks.workunit.client.0.trial125.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T22:23:53.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T22:23:53.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 3 )) 2026-02-25T22:23:53.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2026-02-25T22:23:54.029 INFO:tasks.workunit.client.0.trial125.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T22:23:54.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T22:23:54.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 3 )) 2026-02-25T22:23:54.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2026-02-25T22:23:54.093 INFO:tasks.workunit.client.0.trial125.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T22:23:54.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T22:23:54.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 3 )) 2026-02-25T22:23:54.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:23:54.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:305: verify_chunk_mapping: rm td/test-erasure-code/COPY 2026-02-25T22:23:54.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2026-02-25T22:23:54.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=remap-pool 2026-02-25T22:23:54.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGremap-pool 2026-02-25T22:23:54.098 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2026-02-25T22:23:54.098 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:23:54.313 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2026-02-25T22:23:54.313 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:23:54.313 INFO:tasks.workunit.client.0.trial125.stderr:9' 2026-02-25T22:23:54.313 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 6 9 2026-02-25T22:23:54.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: osds=('4' '6' '9') 2026-02-25T22:23:54.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: local -a osds 2026-02-25T22:23:54.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:308: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2026-02-25T22:23:54.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2026-02-25T22:23:54.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:23:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=6 2026-02-25T22:23:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:23:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2026-02-25T22:23:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-25T22:23:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:23:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=6 2026-02-25T22:23:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:23:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2026-02-25T22:23:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:308: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2026-02-25T22:23:54.316 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:23:54.316 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:23:54.317 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:23:54.317 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:23:54.317 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:23:54.622 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:23:54.622 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2026-02-25T22:23:54.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-25T22:23:54.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:23:54.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=6 2026-02-25T22:23:54.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:23:54.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2026-02-25T22:23:54.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2026-02-25T22:23:54.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2026-02-25T22:23:54.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2026-02-25T22:23:54.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:23:54.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=6 2026-02-25T22:23:54.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:23:54.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/6 2026-02-25T22:23:54.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:23:54.836 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:23:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:23:54.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/6 2026-02-25T22:23:54.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.6 2026-02-25T22:23:54.839 INFO:tasks.workunit.client.0.trial125.stderr:start osd.6 2026-02-25T22:23:54.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 6 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:23:54.839 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/6/whoami 2026-02-25T22:23:54.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 6 = 6 ']' 2026-02-25T22:23:54.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:23:54.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:23:54.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:23:54.859 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:54.855+0000 7fbf064d5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:54.861 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:54.859+0000 7fbf064d5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:54.862 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:54.859+0000 7fbf064d5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:23:55.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 6 2026-02-25T22:23:55.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:23:55.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T22:23:55.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:23:55.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:23:55.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:55.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:23:55.051 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:23:55.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:55.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:23:55.210 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:55.207+0000 7fbf064d5a40 -1 Falling back to public interface 2026-02-25T22:23:55.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:23:55.899 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:23:55.895+0000 7fbf064d5a40 -1 osd.6 84 log_to_monitors true 2026-02-25T22:23:56.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:23:56.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:23:56.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:23:56.264 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:23:56.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:23:56.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:23:56.475 INFO:tasks.workunit.client.0.trial125.stderr:osd.6 up in weight 1 up_from 87 up_thru 87 down_at 84 last_clean_interval [31,84) [v2:127.0.0.1:6850/3065055549,v1:127.0.0.1:6851/3065055549] [v2:127.0.0.1:6852/3065055549,v1:127.0.0.1:6853/3065055549] exists,up 5184d0a8-0efa-4a82-85e6-850f0226d9db 2026-02-25T22:23:56.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:23:56.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:23:56.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:23:56.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:23:56.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:23:56.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:23:56.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:23:56.476 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:23:56.476 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:23:56.477 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:23:56.477 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:23:56.477 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:23:56.477 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:23:56.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:23:56.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:23:56.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:23:56.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:23:56.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:23:56.573 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:23:56.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:23:56.792 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:23:56.792 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:23:56.792 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:23:56.792 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:23:56.792 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:23:56.792 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:23:56.792 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:23:56.792 INFO:tasks.workunit.client.0.trial125.stderr:8 2026-02-25T22:23:56.792 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:23:56.793 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:23:56.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:23:56.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:56.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:23:56.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836506 2026-02-25T22:23:56.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836506 2026-02-25T22:23:56.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506' 2026-02-25T22:23:56.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:56.874 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:23:56.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672985 2026-02-25T22:23:56.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672985 2026-02-25T22:23:56.954 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-42949672985' 2026-02-25T22:23:56.954 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:56.954 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:23:57.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509464 2026-02-25T22:23:57.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509464 2026-02-25T22:23:57.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509464' 2026-02-25T22:23:57.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:57.032 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:23:57.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743434 2026-02-25T22:23:57.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743434 2026-02-25T22:23:57.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509464 3-296352743434' 2026-02-25T22:23:57.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:57.112 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:23:57.189 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247831 2026-02-25T22:23:57.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247831 2026-02-25T22:23:57.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509464 3-296352743434 4-98784247831' 2026-02-25T22:23:57.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:57.190 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:23:57.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514503 2026-02-25T22:23:57.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514503 2026-02-25T22:23:57.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509464 3-296352743434 4-98784247831 5-326417514503' 2026-02-25T22:23:57.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:57.266 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:23:57.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154755 2026-02-25T22:23:57.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154755 2026-02-25T22:23:57.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509464 3-296352743434 4-98784247831 5-326417514503 6-373662154755' 2026-02-25T22:23:57.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:57.345 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T22:23:57.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822676 2026-02-25T22:23:57.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822676 2026-02-25T22:23:57.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509464 3-296352743434 4-98784247831 5-326417514503 6-373662154755 7-154618822676' 2026-02-25T22:23:57.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:57.422 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T22:23:57.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691859 2026-02-25T22:23:57.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691859 2026-02-25T22:23:57.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509464 3-296352743434 4-98784247831 5-326417514503 6-373662154755 7-154618822676 8-171798691859' 2026-02-25T22:23:57.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:57.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T22:23:57.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561044 2026-02-25T22:23:57.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561044 2026-02-25T22:23:57.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509464 3-296352743434 4-98784247831 5-326417514503 6-373662154755 7-154618822676 8-171798691859 9-188978561044' 2026-02-25T22:23:57.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:57.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T22:23:57.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397522 2026-02-25T22:23:57.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397522 2026-02-25T22:23:57.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509464 3-296352743434 4-98784247831 5-326417514503 6-373662154755 7-154618822676 8-171798691859 9-188978561044 10-210453397522' 2026-02-25T22:23:57.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:57.659 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836506 2026-02-25T22:23:57.659 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:57.660 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:23:57.661 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836506 2026-02-25T22:23:57.661 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:57.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836506 2026-02-25T22:23:57.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2026-02-25T22:23:57.662 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 21474836506 2026-02-25T22:23:57.663 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:23:57.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836503 -lt 21474836506 2026-02-25T22:23:57.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:23:58.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:23:58.878 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:23:59.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836506 -lt 21474836506 2026-02-25T22:23:59.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:59.087 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672985 2026-02-25T22:23:59.088 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:59.089 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:23:59.089 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672985 2026-02-25T22:23:59.089 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:59.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672985 2026-02-25T22:23:59.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2026-02-25T22:23:59.091 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 42949672985 2026-02-25T22:23:59.091 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:23:59.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672985 -lt 42949672985 2026-02-25T22:23:59.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:59.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-64424509464 2026-02-25T22:23:59.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:59.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:23:59.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509464 2026-02-25T22:23:59.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:59.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=64424509464 2026-02-25T22:23:59.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2026-02-25T22:23:59.312 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 64424509464 2026-02-25T22:23:59.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:23:59.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 64424509464 -lt 64424509464 2026-02-25T22:23:59.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:59.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-296352743434 2026-02-25T22:23:59.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:59.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:23:59.540 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-296352743434 2026-02-25T22:23:59.540 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:59.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=296352743434 2026-02-25T22:23:59.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 296352743434' 2026-02-25T22:23:59.541 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 296352743434 2026-02-25T22:23:59.542 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:23:59.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 296352743434 -lt 296352743434 2026-02-25T22:23:59.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:59.755 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247831 2026-02-25T22:23:59.755 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:59.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:23:59.757 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247831 2026-02-25T22:23:59.757 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:59.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247831 2026-02-25T22:23:59.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247831' 2026-02-25T22:23:59.758 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.4 seq 98784247831 2026-02-25T22:23:59.758 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:23:59.977 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247831 -lt 98784247831 2026-02-25T22:23:59.977 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:59.977 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-326417514503 2026-02-25T22:23:59.978 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:59.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:23:59.979 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-326417514503 2026-02-25T22:23:59.979 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:59.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=326417514503 2026-02-25T22:23:59.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 326417514503' 2026-02-25T22:23:59.981 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.5 seq 326417514503 2026-02-25T22:23:59.981 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:24:00.197 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 326417514503 -lt 326417514503 2026-02-25T22:24:00.197 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:00.198 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-373662154755 2026-02-25T22:24:00.198 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:00.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:24:00.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-373662154755 2026-02-25T22:24:00.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:00.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=373662154755 2026-02-25T22:24:00.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 373662154755' 2026-02-25T22:24:00.201 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.6 seq 373662154755 2026-02-25T22:24:00.202 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:24:00.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 373662154755 -lt 373662154755 2026-02-25T22:24:00.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:00.412 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-154618822676 2026-02-25T22:24:00.412 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:00.413 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T22:24:00.413 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822676 2026-02-25T22:24:00.413 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:00.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=154618822676 2026-02-25T22:24:00.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 154618822676' 2026-02-25T22:24:00.415 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.7 seq 154618822676 2026-02-25T22:24:00.415 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T22:24:00.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 154618822677 -lt 154618822676 2026-02-25T22:24:00.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:00.638 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-171798691859 2026-02-25T22:24:00.638 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:00.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T22:24:00.640 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691859 2026-02-25T22:24:00.640 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:00.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691859 2026-02-25T22:24:00.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 171798691859' 2026-02-25T22:24:00.642 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.8 seq 171798691859 2026-02-25T22:24:00.642 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T22:24:00.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691860 -lt 171798691859 2026-02-25T22:24:00.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:00.847 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-188978561044 2026-02-25T22:24:00.848 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:00.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T22:24:00.849 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-188978561044 2026-02-25T22:24:00.850 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:00.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561044 2026-02-25T22:24:00.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 188978561044' 2026-02-25T22:24:00.851 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.9 seq 188978561044 2026-02-25T22:24:00.851 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T22:24:01.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561044 -lt 188978561044 2026-02-25T22:24:01.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:01.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-210453397522 2026-02-25T22:24:01.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:01.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T22:24:01.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-210453397522 2026-02-25T22:24:01.065 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:01.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397522 2026-02-25T22:24:01.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 210453397522' 2026-02-25T22:24:01.066 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.10 seq 210453397522 2026-02-25T22:24:01.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T22:24:01.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397522 -lt 210453397522 2026-02-25T22:24:01.281 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:24:01.281 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:24:01.281 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:24:01.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 28 == 0 2026-02-25T22:24:01.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:24:01.567 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:24:01.568 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:24:01.568 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:24:01.568 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:24:01.568 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:24:01.568 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:24:01.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=28 2026-02-25T22:24:01.778 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:24:01.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:24:01.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:24:02.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 = 28 2026-02-25T22:24:02.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:24:02.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:24:02.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:309: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2026-02-25T22:24:02.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:309: verify_chunk_mapping: grep --quiet SECONDremap-pool 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:24:02.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:24:02.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:24:02.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:24:02.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:24:02.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:24:02.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2026-02-25T22:24:02.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-25T22:24:02.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:24:02.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-25T22:24:02.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:24:02.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2026-02-25T22:24:02.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2026-02-25T22:24:02.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2026-02-25T22:24:02.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2026-02-25T22:24:02.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:24:02.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-25T22:24:02.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:24:02.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/9 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:24:02.563 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:24:02.564 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:02.564 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:24:02.564 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:24:02.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' 2026-02-25T22:24:02.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:24:02.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T22:24:02.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T22:24:02.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:24:02.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:24:02.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:24:02.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:24:02.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:24:02.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:24:02.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/9 2026-02-25T22:24:02.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-25T22:24:02.566 INFO:tasks.workunit.client.0.trial125.stderr:start osd.9 2026-02-25T22:24:02.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=02790f30-b792-4252-9c7b-b7a0ad712049 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.45907/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:24:02.566 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/9/whoami 2026-02-25T22:24:02.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-25T22:24:02.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:24:02.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:24:02.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:24:02.581 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:24:02.579+0000 7f859500da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:02.583 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:24:02.579+0000 7f859500da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:02.584 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:24:02.579+0000 7f859500da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:02.776 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-25T22:24:02.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:24:02.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-25T22:24:02.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:24:02.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:24:02.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:24:02.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:24:02.777 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:24:02.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:24:02.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-25T22:24:02.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:24:03.034 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:24:03.031+0000 7f859500da40 -1 Falling back to public interface 2026-02-25T22:24:03.687 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:24:03.683+0000 7f859500da40 -1 osd.9 89 log_to_monitors true 2026-02-25T22:24:03.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:24:03.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:24:03.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:24:03.989 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:24:03.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:24:03.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-25T22:24:04.200 INFO:tasks.workunit.client.0.trial125.stderr:osd.9 up in weight 1 up_from 92 up_thru 92 down_at 89 last_clean_interval [44,89) [v2:127.0.0.1:6874/3542765967,v1:127.0.0.1:6875/3542765967] [v2:127.0.0.1:6876/3542765967,v1:127.0.0.1:6877/3542765967] exists,up 0d0c8d03-6881-4e3b-9477-1d0d8e85e7b0 2026-02-25T22:24:04.200 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:24:04.200 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:24:04.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:24:04.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:24:04.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:24:04.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:24:04.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:24:04.202 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:24:04.202 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:24:04.202 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:24:04.202 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:24:04.202 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:24:04.202 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:24:04.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:24:04.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:24:04.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:24:04.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:24:04.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:24:04.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:24:04.519 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:24:04.519 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:24:04.520 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:24:04.520 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:24:04.520 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:24:04.520 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:24:04.520 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:24:04.520 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:24:04.520 INFO:tasks.workunit.client.0.trial125.stderr:8 2026-02-25T22:24:04.520 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:04.520 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:24:04.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:24:04.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:04.521 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:24:04.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836509 2026-02-25T22:24:04.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836509 2026-02-25T22:24:04.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509' 2026-02-25T22:24:04.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:04.604 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:24:04.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672988 2026-02-25T22:24:04.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672988 2026-02-25T22:24:04.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-42949672988' 2026-02-25T22:24:04.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:04.685 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:24:04.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509468 2026-02-25T22:24:04.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509468 2026-02-25T22:24:04.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468' 2026-02-25T22:24:04.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:04.764 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:24:04.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743438 2026-02-25T22:24:04.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743438 2026-02-25T22:24:04.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-296352743438' 2026-02-25T22:24:04.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:04.841 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:24:04.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247834 2026-02-25T22:24:04.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247834 2026-02-25T22:24:04.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-296352743438 4-98784247834' 2026-02-25T22:24:04.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:04.921 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:24:04.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514506 2026-02-25T22:24:04.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514506 2026-02-25T22:24:04.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-296352743438 4-98784247834 5-326417514506' 2026-02-25T22:24:04.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:04.998 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:24:05.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154758 2026-02-25T22:24:05.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154758 2026-02-25T22:24:05.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-296352743438 4-98784247834 5-326417514506 6-373662154758' 2026-02-25T22:24:05.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:05.075 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T22:24:05.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822680 2026-02-25T22:24:05.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822680 2026-02-25T22:24:05.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-296352743438 4-98784247834 5-326417514506 6-373662154758 7-154618822680' 2026-02-25T22:24:05.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:05.157 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T22:24:05.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691863 2026-02-25T22:24:05.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691863 2026-02-25T22:24:05.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-296352743438 4-98784247834 5-326417514506 6-373662154758 7-154618822680 8-171798691863' 2026-02-25T22:24:05.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:05.233 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T22:24:05.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991235 2026-02-25T22:24:05.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991235 2026-02-25T22:24:05.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-296352743438 4-98784247834 5-326417514506 6-373662154758 7-154618822680 8-171798691863 9-395136991235' 2026-02-25T22:24:05.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:05.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T22:24:05.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397525 2026-02-25T22:24:05.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397525 2026-02-25T22:24:05.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-296352743438 4-98784247834 5-326417514506 6-373662154758 7-154618822680 8-171798691863 9-395136991235 10-210453397525' 2026-02-25T22:24:05.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:05.389 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836509 2026-02-25T22:24:05.389 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:05.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:24:05.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836509 2026-02-25T22:24:05.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:05.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836509 2026-02-25T22:24:05.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-25T22:24:05.392 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 21474836509 2026-02-25T22:24:05.393 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:24:05.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836509 -lt 21474836509 2026-02-25T22:24:05.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:05.617 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672988 2026-02-25T22:24:05.617 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:05.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:24:05.618 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672988 2026-02-25T22:24:05.618 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:05.620 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672988 2026-02-25T22:24:05.620 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2026-02-25T22:24:05.620 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 42949672988 2026-02-25T22:24:05.620 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:24:05.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672986 -lt 42949672988 2026-02-25T22:24:05.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:24:06.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:24:06.850 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:24:07.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672989 -lt 42949672988 2026-02-25T22:24:07.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:07.059 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-64424509468 2026-02-25T22:24:07.059 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:07.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:24:07.061 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509468 2026-02-25T22:24:07.061 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:07.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=64424509468 2026-02-25T22:24:07.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2026-02-25T22:24:07.062 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 64424509468 2026-02-25T22:24:07.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:24:07.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 64424509468 -lt 64424509468 2026-02-25T22:24:07.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:07.287 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-296352743438 2026-02-25T22:24:07.287 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:07.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:24:07.288 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-296352743438 2026-02-25T22:24:07.288 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:07.290 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=296352743438 2026-02-25T22:24:07.290 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 296352743438' 2026-02-25T22:24:07.290 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 296352743438 2026-02-25T22:24:07.290 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:24:07.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 296352743438 -lt 296352743438 2026-02-25T22:24:07.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:07.495 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247834 2026-02-25T22:24:07.495 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:07.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:24:07.497 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247834 2026-02-25T22:24:07.497 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:07.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247834 2026-02-25T22:24:07.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247834' 2026-02-25T22:24:07.499 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.4 seq 98784247834 2026-02-25T22:24:07.499 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:24:07.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247835 -lt 98784247834 2026-02-25T22:24:07.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:07.710 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-326417514506 2026-02-25T22:24:07.710 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:07.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:24:07.712 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-326417514506 2026-02-25T22:24:07.712 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:07.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=326417514506 2026-02-25T22:24:07.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 326417514506' 2026-02-25T22:24:07.713 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.5 seq 326417514506 2026-02-25T22:24:07.714 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:24:07.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 326417514506 -lt 326417514506 2026-02-25T22:24:07.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:07.930 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-373662154758 2026-02-25T22:24:07.930 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:07.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:24:07.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-373662154758 2026-02-25T22:24:07.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:07.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=373662154758 2026-02-25T22:24:07.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 373662154758' 2026-02-25T22:24:07.934 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.6 seq 373662154758 2026-02-25T22:24:07.934 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:24:08.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 373662154759 -lt 373662154758 2026-02-25T22:24:08.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:08.142 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-154618822680 2026-02-25T22:24:08.142 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:08.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T22:24:08.143 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822680 2026-02-25T22:24:08.143 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:08.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=154618822680 2026-02-25T22:24:08.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 154618822680' 2026-02-25T22:24:08.145 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.7 seq 154618822680 2026-02-25T22:24:08.145 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T22:24:08.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 154618822680 -lt 154618822680 2026-02-25T22:24:08.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:08.360 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-171798691863 2026-02-25T22:24:08.360 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:08.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T22:24:08.362 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691863 2026-02-25T22:24:08.362 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:08.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691863 2026-02-25T22:24:08.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 171798691863' 2026-02-25T22:24:08.363 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.8 seq 171798691863 2026-02-25T22:24:08.364 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T22:24:08.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691863 -lt 171798691863 2026-02-25T22:24:08.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:08.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-395136991235 2026-02-25T22:24:08.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:08.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T22:24:08.584 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-395136991235 2026-02-25T22:24:08.585 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:08.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=395136991235 2026-02-25T22:24:08.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 395136991235' 2026-02-25T22:24:08.586 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.9 seq 395136991235 2026-02-25T22:24:08.586 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T22:24:08.808 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 395136991235 -lt 395136991235 2026-02-25T22:24:08.808 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:08.809 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-210453397525 2026-02-25T22:24:08.809 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:08.810 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T22:24:08.811 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-210453397525 2026-02-25T22:24:08.811 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:08.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397525 2026-02-25T22:24:08.813 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 210453397525' 2026-02-25T22:24:08.813 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.10 seq 210453397525 2026-02-25T22:24:08.813 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T22:24:09.027 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397526 -lt 210453397525 2026-02-25T22:24:09.027 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:24:09.028 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:24:09.028 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:24:09.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 28 == 0 2026-02-25T22:24:09.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:24:09.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:24:09.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:24:09.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:24:09.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:24:09.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:24:09.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:24:09.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=28 2026-02-25T22:24:09.525 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:24:09.526 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:24:09.526 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:24:09.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 = 28 2026-02-25T22:24:09.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:24:09.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:24:09.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:338: TEST_chunk_mapping: delete_pool remap-pool 2026-02-25T22:24:09.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=remap-pool 2026-02-25T22:24:09.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2026-02-25T22:24:10.021 INFO:tasks.workunit.client.0.trial125.stderr:pool 'remap-pool' does not exist 2026-02-25T22:24:10.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:339: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2026-02-25T22:24:10.307 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile remap-profile does not exist 2026-02-25T22:24:10.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T22:24:10.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2026-02-25T22:24:10.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2026-02-25T22:24:10.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: erasure_code_plugin_exists: local plugin=isa 2026-02-25T22:24:10.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: erasure_code_plugin_exists: local status 2026-02-25T22:24:10.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: erasure_code_plugin_exists: local grepstr 2026-02-25T22:24:10.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: erasure_code_plugin_exists: local s 2026-02-25T22:24:10.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: case `uname` in 2026-02-25T22:24:10.320 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: uname 2026-02-25T22:24:10.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: erasure_code_plugin_exists: grepstr='isa.*No such file' 2026-02-25T22:24:10.321 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2026-02-25T22:24:10.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: s= 2026-02-25T22:24:10.594 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: erasure_code_plugin_exists: local status=0 2026-02-25T22:24:10.594 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2026-02-25T22:24:10.594 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2026-02-25T22:24:10.886 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile TESTPROFILE does not exist 2026-02-25T22:24:10.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: erasure_code_plugin_exists: return 0 2026-02-25T22:24:10.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:197: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2026-02-25T22:24:10.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:198: TEST_rados_put_get_isa: local poolname=pool-isa 2026-02-25T22:24:10.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:200: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2026-02-25T22:24:11.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:203: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2026-02-25T22:24:11.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2026-02-25T22:24:11.540 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-isa' already exists 2026-02-25T22:24:11.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:24:12.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:206: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2026-02-25T22:24:12.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T22:24:12.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2026-02-25T22:24:12.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T22:24:12.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:12.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T22:24:12.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:12.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T22:24:12.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:12.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T22:24:12.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:12.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T22:24:12.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T22:24:12.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:24:12.598 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:24:12.599 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T22:24:12.601 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2026-02-25T22:24:12.601 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-isa 2026-02-25T22:24:12.601 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:12.602 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-isa SOMETHING 2026-02-25T22:24:12.602 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:12.815 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2026-02-25T22:24:12.815 INFO:tasks.workunit.client.0.trial125.stderr:10 2026-02-25T22:24:12.815 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:24:12.816 INFO:tasks.workunit.client.0.trial125.stderr:7' 2026-02-25T22:24:12.816 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 10 0 7 2026-02-25T22:24:12.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('4' '10' '0' '7') 2026-02-25T22:24:12.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T22:24:12.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2026-02-25T22:24:12.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2026-02-25T22:24:13.079 INFO:tasks.workunit.client.0.trial125.stderr:osd.7 is already out. 2026-02-25T22:24:13.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T22:24:18.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-isa SOMETHING 2026-02-25T22:24:18.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-isa 2026-02-25T22:24:18.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:18.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2026-02-25T22:24:18.092 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-isa SOMETHING 2026-02-25T22:24:18.092 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:18.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2026-02-25T22:24:18.307 INFO:tasks.workunit.client.0.trial125.stderr:10 2026-02-25T22:24:18.307 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:24:18.307 INFO:tasks.workunit.client.0.trial125.stderr:1' 2026-02-25T22:24:18.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 10 0 1 2026-02-25T22:24:18.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:24:18.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:24:18.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2026-02-25T22:24:18.555 INFO:tasks.workunit.client.0.trial125.stderr:osd.7 is already in. 2026-02-25T22:24:18.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T22:24:18.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_isa: delete_pool pool-isa 2026-02-25T22:24:18.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-isa 2026-02-25T22:24:18.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2026-02-25T22:24:18.858 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-isa' does not exist 2026-02-25T22:24:18.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T22:24:18.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2026-02-25T22:24:18.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2026-02-25T22:24:18.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:214: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2026-02-25T22:24:18.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T22:24:18.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2026-02-25T22:24:18.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T22:24:18.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:18.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T22:24:18.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:18.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T22:24:18.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:18.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T22:24:18.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:18.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T22:24:18.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T22:24:18.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:24:18.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:24:18.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T22:24:18.923 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2026-02-25T22:24:18.923 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-25T22:24:18.923 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:18.923 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-25T22:24:18.923 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:19.138 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=6 2026-02-25T22:24:19.138 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:24:19.138 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:24:19.138 INFO:tasks.workunit.client.0.trial125.stderr:9' 2026-02-25T22:24:19.138 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 6 0 7 9 2026-02-25T22:24:19.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('6' '0' '7' '9') 2026-02-25T22:24:19.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T22:24:19.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2026-02-25T22:24:19.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2026-02-25T22:24:19.385 INFO:tasks.workunit.client.0.trial125.stderr:osd.9 is already out. 2026-02-25T22:24:19.395 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T22:24:24.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds ecpool SOMETHING 2026-02-25T22:24:24.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-25T22:24:24.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:24.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2026-02-25T22:24:24.398 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-25T22:24:24.398 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:24.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=6 2026-02-25T22:24:24.611 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:24:24.612 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:24:24.612 INFO:tasks.workunit.client.0.trial125.stderr:5' 2026-02-25T22:24:24.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 6 0 7 5 2026-02-25T22:24:24.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:24:24.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:24:24.631 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2026-02-25T22:24:24.859 INFO:tasks.workunit.client.0.trial125.stderr:osd.9 is already in. 2026-02-25T22:24:24.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T22:24:24.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:216: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2026-02-25T22:24:24.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:217: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2026-02-25T22:24:24.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2026-02-25T22:24:25.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2026-02-25T22:24:25.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2026-02-25T22:24:25.533 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:24:25.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:24:26.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2026-02-25T22:24:26.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T22:24:26.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2026-02-25T22:24:26.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T22:24:26.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:26.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T22:24:26.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:26.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T22:24:26.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:26.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T22:24:26.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:26.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T22:24:26.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T22:24:26.569 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:24:26.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:24:26.589 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T22:24:26.591 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2026-02-25T22:24:26.591 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:24:26.591 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:26.591 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T22:24:26.591 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:26.796 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T22:24:26.796 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:24:26.796 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:26.796 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:24:26.796 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:24:26.796 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:24:26.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2026-02-25T22:24:26.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '1' '9' '4' '3' '10') 2026-02-25T22:24:26.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T22:24:26.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2026-02-25T22:24:26.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2026-02-25T22:24:27.058 INFO:tasks.workunit.client.0.trial125.stderr:osd.10 is already out. 2026-02-25T22:24:27.070 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T22:24:32.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-jerasure SOMETHING 2026-02-25T22:24:32.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:24:32.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:32.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<10\>' 2026-02-25T22:24:32.073 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T22:24:32.073 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:32.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T22:24:32.284 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:24:32.284 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:32.284 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:24:32.284 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:24:32.284 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:24:32.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 6 2026-02-25T22:24:32.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:24:32.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:24:32.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 10 2026-02-25T22:24:32.532 INFO:tasks.workunit.client.0.trial125.stderr:osd.10 is already in. 2026-02-25T22:24:32.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T22:24:32.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:227: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2026-02-25T22:24:32.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:104: rados_osds_out_in: local dir=td/test-erasure-code 2026-02-25T22:24:32.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: local poolname=pool-jerasure 2026-02-25T22:24:32.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: local objname=SOMETHING 2026-02-25T22:24:32.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-25T22:24:32.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 FFFF 2026-02-25T22:24:32.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-25T22:24:32.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 GGGG 2026-02-25T22:24:32.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-25T22:24:32.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 HHHH 2026-02-25T22:24:32.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-25T22:24:32.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 IIII 2026-02-25T22:24:32.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:116: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T22:24:32.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:117: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:24:32.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:118: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:24:32.624 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:119: rados_osds_out_in: rm td/test-erasure-code/COPY 2026-02-25T22:24:32.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: wait_for_clean 2026-02-25T22:24:32.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:24:32.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:24:32.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:24:32.626 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:24:32.626 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:24:32.626 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:24:32.627 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:24:32.627 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:24:32.627 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:24:32.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:24:32.734 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:24:32.734 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:24:32.734 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:24:32.734 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:24:32.734 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:24:32.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:8 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:24:32.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:32.948 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:24:33.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836517 2026-02-25T22:24:33.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836517 2026-02-25T22:24:33.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517' 2026-02-25T22:24:33.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:33.025 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:24:33.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672996 2026-02-25T22:24:33.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672996 2026-02-25T22:24:33.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-42949672996' 2026-02-25T22:24:33.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:33.104 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:24:33.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509475 2026-02-25T22:24:33.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509475 2026-02-25T22:24:33.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509475' 2026-02-25T22:24:33.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:33.181 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:24:33.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743445 2026-02-25T22:24:33.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743445 2026-02-25T22:24:33.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509475 3-296352743445' 2026-02-25T22:24:33.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:33.258 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:24:33.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247842 2026-02-25T22:24:33.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247842 2026-02-25T22:24:33.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509475 3-296352743445 4-98784247842' 2026-02-25T22:24:33.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:33.340 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:24:33.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514514 2026-02-25T22:24:33.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514514 2026-02-25T22:24:33.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509475 3-296352743445 4-98784247842 5-326417514514' 2026-02-25T22:24:33.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:33.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:24:33.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154766 2026-02-25T22:24:33.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154766 2026-02-25T22:24:33.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509475 3-296352743445 4-98784247842 5-326417514514 6-373662154766' 2026-02-25T22:24:33.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:33.498 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T22:24:33.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822687 2026-02-25T22:24:33.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822687 2026-02-25T22:24:33.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509475 3-296352743445 4-98784247842 5-326417514514 6-373662154766 7-154618822687' 2026-02-25T22:24:33.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:33.576 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T22:24:33.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691871 2026-02-25T22:24:33.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691871 2026-02-25T22:24:33.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509475 3-296352743445 4-98784247842 5-326417514514 6-373662154766 7-154618822687 8-171798691871' 2026-02-25T22:24:33.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:33.654 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T22:24:33.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991242 2026-02-25T22:24:33.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991242 2026-02-25T22:24:33.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509475 3-296352743445 4-98784247842 5-326417514514 6-373662154766 7-154618822687 8-171798691871 9-395136991242' 2026-02-25T22:24:33.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:33.730 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T22:24:33.804 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397533 2026-02-25T22:24:33.804 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397533 2026-02-25T22:24:33.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-42949672996 2-64424509475 3-296352743445 4-98784247842 5-326417514514 6-373662154766 7-154618822687 8-171798691871 9-395136991242 10-210453397533' 2026-02-25T22:24:33.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:33.805 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836517 2026-02-25T22:24:33.805 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:33.806 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:24:33.807 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836517 2026-02-25T22:24:33.807 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:33.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836517 2026-02-25T22:24:33.809 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 21474836517 2026-02-25T22:24:33.810 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2026-02-25T22:24:33.810 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:24:34.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836515 -lt 21474836517 2026-02-25T22:24:34.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:24:35.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:24:35.019 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:24:35.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836517 -lt 21474836517 2026-02-25T22:24:35.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:35.233 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672996 2026-02-25T22:24:35.233 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:35.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:24:35.235 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672996 2026-02-25T22:24:35.235 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:35.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672996 2026-02-25T22:24:35.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2026-02-25T22:24:35.236 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 42949672996 2026-02-25T22:24:35.237 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:24:35.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672996 -lt 42949672996 2026-02-25T22:24:35.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:35.442 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-64424509475 2026-02-25T22:24:35.443 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:35.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:24:35.445 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509475 2026-02-25T22:24:35.445 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:35.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=64424509475 2026-02-25T22:24:35.446 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 64424509475 2026-02-25T22:24:35.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2026-02-25T22:24:35.447 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:24:35.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 64424509476 -lt 64424509475 2026-02-25T22:24:35.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:35.659 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-296352743445 2026-02-25T22:24:35.659 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:35.660 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:24:35.661 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-296352743445 2026-02-25T22:24:35.661 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:35.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=296352743445 2026-02-25T22:24:35.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 296352743445' 2026-02-25T22:24:35.663 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 296352743445 2026-02-25T22:24:35.663 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:24:35.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 296352743446 -lt 296352743445 2026-02-25T22:24:35.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:35.875 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247842 2026-02-25T22:24:35.875 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:35.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:24:35.877 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247842 2026-02-25T22:24:35.878 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:35.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247842 2026-02-25T22:24:35.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247842' 2026-02-25T22:24:35.879 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 98784247842 2026-02-25T22:24:35.880 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:24:36.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247842 -lt 98784247842 2026-02-25T22:24:36.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:36.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-326417514514 2026-02-25T22:24:36.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:36.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:24:36.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-326417514514 2026-02-25T22:24:36.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:36.098 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=326417514514 2026-02-25T22:24:36.099 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 326417514514 2026-02-25T22:24:36.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 326417514514' 2026-02-25T22:24:36.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:24:36.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 326417514514 -lt 326417514514 2026-02-25T22:24:36.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:36.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-373662154766 2026-02-25T22:24:36.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:36.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:24:36.314 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-373662154766 2026-02-25T22:24:36.314 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:36.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=373662154766 2026-02-25T22:24:36.316 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 373662154766 2026-02-25T22:24:36.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 373662154766' 2026-02-25T22:24:36.316 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:24:36.531 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 373662154766 -lt 373662154766 2026-02-25T22:24:36.531 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:36.532 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-154618822687 2026-02-25T22:24:36.532 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:36.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T22:24:36.534 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822687 2026-02-25T22:24:36.534 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:36.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=154618822687 2026-02-25T22:24:36.536 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.7 seq 154618822687 2026-02-25T22:24:36.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 154618822687' 2026-02-25T22:24:36.537 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T22:24:36.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 154618822688 -lt 154618822687 2026-02-25T22:24:36.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:36.755 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-171798691871 2026-02-25T22:24:36.755 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:36.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T22:24:36.757 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691871 2026-02-25T22:24:36.757 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:36.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691871 2026-02-25T22:24:36.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 171798691871' 2026-02-25T22:24:36.759 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.8 seq 171798691871 2026-02-25T22:24:36.759 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T22:24:36.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691871 -lt 171798691871 2026-02-25T22:24:36.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:36.976 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-395136991242 2026-02-25T22:24:36.976 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:36.977 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T22:24:36.978 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-395136991242 2026-02-25T22:24:36.978 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:36.979 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=395136991242 2026-02-25T22:24:36.980 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 395136991242' 2026-02-25T22:24:36.980 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.9 seq 395136991242 2026-02-25T22:24:36.980 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T22:24:37.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 395136991243 -lt 395136991242 2026-02-25T22:24:37.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:37.195 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-210453397533 2026-02-25T22:24:37.195 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:37.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T22:24:37.197 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-210453397533 2026-02-25T22:24:37.197 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:37.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397533 2026-02-25T22:24:37.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 210453397533' 2026-02-25T22:24:37.199 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.10 seq 210453397533 2026-02-25T22:24:37.199 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T22:24:37.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397534 -lt 210453397533 2026-02-25T22:24:37.422 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:24:37.423 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:24:37.423 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:24:37.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 28 == 0 2026-02-25T22:24:37.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:24:37.702 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:24:37.702 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:24:37.702 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:24:37.702 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:24:37.702 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:24:37.703 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:24:37.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=28 2026-02-25T22:24:37.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:24:37.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:24:37.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:24:38.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 = 28 2026-02-25T22:24:38.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:24:38.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:24:38.183 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-25T22:24:38.183 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:24:38.183 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:38.183 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T22:24:38.183 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:38.395 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T22:24:38.396 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:24:38.396 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:38.396 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:24:38.396 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:24:38.396 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:24:38.396 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2026-02-25T22:24:38.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2026-02-25T22:24:38.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: osds=('5' '1' '9' '4' '3' '10') 2026-02-25T22:24:38.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: local -a osds 2026-02-25T22:24:38.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2026-02-25T22:24:38.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 5 2026-02-25T22:24:38.700 INFO:tasks.workunit.client.0.trial125.stderr:osd.5 is already out. 2026-02-25T22:24:38.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2026-02-25T22:24:38.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 1 2026-02-25T22:24:39.006 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 is already out. 2026-02-25T22:24:39.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: wait_for_clean 2026-02-25T22:24:39.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:24:39.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:24:39.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:24:39.018 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:24:39.018 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:24:39.018 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:24:39.018 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:24:39.018 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:24:39.018 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:24:39.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:24:39.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:24:39.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:24:39.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:24:39.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:24:39.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:24:39.328 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:24:39.328 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:24:39.328 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:24:39.328 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:24:39.329 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:24:39.329 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:24:39.329 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:24:39.329 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:24:39.329 INFO:tasks.workunit.client.0.trial125.stderr:8 2026-02-25T22:24:39.329 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:39.329 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:24:39.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:24:39.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:39.330 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:24:39.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836520 2026-02-25T22:24:39.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836520 2026-02-25T22:24:39.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520' 2026-02-25T22:24:39.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:39.408 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:24:39.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672999 2026-02-25T22:24:39.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672999 2026-02-25T22:24:39.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-42949672999' 2026-02-25T22:24:39.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:39.488 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:24:39.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509479 2026-02-25T22:24:39.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509479 2026-02-25T22:24:39.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479' 2026-02-25T22:24:39.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:39.568 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:24:39.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743449 2026-02-25T22:24:39.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743449 2026-02-25T22:24:39.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479 3-296352743449' 2026-02-25T22:24:39.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:39.646 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:24:39.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247845 2026-02-25T22:24:39.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247845 2026-02-25T22:24:39.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479 3-296352743449 4-98784247845' 2026-02-25T22:24:39.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:39.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:24:39.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514517 2026-02-25T22:24:39.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514517 2026-02-25T22:24:39.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479 3-296352743449 4-98784247845 5-326417514517' 2026-02-25T22:24:39.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:39.804 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:24:39.883 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154769 2026-02-25T22:24:39.883 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154769 2026-02-25T22:24:39.884 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479 3-296352743449 4-98784247845 5-326417514517 6-373662154769' 2026-02-25T22:24:39.884 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:39.884 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T22:24:39.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822691 2026-02-25T22:24:39.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822691 2026-02-25T22:24:39.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479 3-296352743449 4-98784247845 5-326417514517 6-373662154769 7-154618822691' 2026-02-25T22:24:39.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:39.961 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T22:24:40.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691874 2026-02-25T22:24:40.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691874 2026-02-25T22:24:40.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479 3-296352743449 4-98784247845 5-326417514517 6-373662154769 7-154618822691 8-171798691874' 2026-02-25T22:24:40.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:40.040 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T22:24:40.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991246 2026-02-25T22:24:40.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991246 2026-02-25T22:24:40.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479 3-296352743449 4-98784247845 5-326417514517 6-373662154769 7-154618822691 8-171798691874 9-395136991246' 2026-02-25T22:24:40.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:40.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T22:24:40.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397536 2026-02-25T22:24:40.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397536 2026-02-25T22:24:40.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-42949672999 2-64424509479 3-296352743449 4-98784247845 5-326417514517 6-373662154769 7-154618822691 8-171798691874 9-395136991246 10-210453397536' 2026-02-25T22:24:40.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:40.198 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836520 2026-02-25T22:24:40.199 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:40.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:24:40.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836520 2026-02-25T22:24:40.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:40.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836520 2026-02-25T22:24:40.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2026-02-25T22:24:40.202 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 21474836520 2026-02-25T22:24:40.202 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:24:40.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836518 -lt 21474836520 2026-02-25T22:24:40.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:24:41.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:24:41.417 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:24:41.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836520 -lt 21474836520 2026-02-25T22:24:41.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:41.627 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672999 2026-02-25T22:24:41.627 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:41.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:24:41.629 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672999 2026-02-25T22:24:41.629 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:41.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672999 2026-02-25T22:24:41.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2026-02-25T22:24:41.630 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 42949672999 2026-02-25T22:24:41.631 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:24:41.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949673000 -lt 42949672999 2026-02-25T22:24:41.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:41.853 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-64424509479 2026-02-25T22:24:41.853 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:41.854 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:24:41.855 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509479 2026-02-25T22:24:41.856 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:41.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=64424509479 2026-02-25T22:24:41.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2026-02-25T22:24:41.857 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 64424509479 2026-02-25T22:24:41.858 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:24:42.079 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 64424509479 -lt 64424509479 2026-02-25T22:24:42.079 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:42.080 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-296352743449 2026-02-25T22:24:42.080 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:42.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:24:42.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-296352743449 2026-02-25T22:24:42.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:42.084 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=296352743449 2026-02-25T22:24:42.084 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 296352743449' 2026-02-25T22:24:42.084 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 296352743449 2026-02-25T22:24:42.084 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:24:42.290 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 296352743449 -lt 296352743449 2026-02-25T22:24:42.290 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:42.291 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247845 2026-02-25T22:24:42.291 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:42.292 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:24:42.293 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247845 2026-02-25T22:24:42.293 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:42.294 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247845 2026-02-25T22:24:42.294 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247845' 2026-02-25T22:24:42.295 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 98784247845 2026-02-25T22:24:42.295 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:24:42.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247845 -lt 98784247845 2026-02-25T22:24:42.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:42.504 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-326417514517 2026-02-25T22:24:42.504 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:42.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:24:42.506 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-326417514517 2026-02-25T22:24:42.506 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:42.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=326417514517 2026-02-25T22:24:42.508 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 326417514517 2026-02-25T22:24:42.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 326417514517' 2026-02-25T22:24:42.508 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:24:42.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 326417514518 -lt 326417514517 2026-02-25T22:24:42.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:42.725 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-373662154769 2026-02-25T22:24:42.725 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:42.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:24:42.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-373662154769 2026-02-25T22:24:42.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:42.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=373662154769 2026-02-25T22:24:42.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 373662154769' 2026-02-25T22:24:42.728 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 373662154769 2026-02-25T22:24:42.729 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:24:42.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 373662154770 -lt 373662154769 2026-02-25T22:24:42.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:42.941 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-154618822691 2026-02-25T22:24:42.942 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:42.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T22:24:42.943 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822691 2026-02-25T22:24:42.944 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:42.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=154618822691 2026-02-25T22:24:42.945 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.7 seq 154618822691 2026-02-25T22:24:42.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 154618822691' 2026-02-25T22:24:42.946 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T22:24:43.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 154618822691 -lt 154618822691 2026-02-25T22:24:43.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:43.171 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-171798691874 2026-02-25T22:24:43.171 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:43.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T22:24:43.173 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691874 2026-02-25T22:24:43.173 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:43.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691874 2026-02-25T22:24:43.175 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.8 seq 171798691874 2026-02-25T22:24:43.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 171798691874' 2026-02-25T22:24:43.175 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T22:24:43.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691874 -lt 171798691874 2026-02-25T22:24:43.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:43.397 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-395136991246 2026-02-25T22:24:43.397 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:43.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T22:24:43.399 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-395136991246 2026-02-25T22:24:43.399 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:43.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=395136991246 2026-02-25T22:24:43.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 395136991246' 2026-02-25T22:24:43.401 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.9 seq 395136991246 2026-02-25T22:24:43.401 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T22:24:43.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 395136991246 -lt 395136991246 2026-02-25T22:24:43.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:43.617 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-210453397536 2026-02-25T22:24:43.617 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:43.619 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T22:24:43.619 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-210453397536 2026-02-25T22:24:43.620 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:43.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397536 2026-02-25T22:24:43.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 210453397536' 2026-02-25T22:24:43.621 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.10 seq 210453397536 2026-02-25T22:24:43.622 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T22:24:43.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397537 -lt 210453397536 2026-02-25T22:24:43.838 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:24:43.838 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:24:43.838 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:24:44.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 28 == 0 2026-02-25T22:24:44.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:24:44.125 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:24:44.126 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:24:44.126 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:24:44.126 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:24:44.126 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:24:44.126 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:24:44.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=28 2026-02-25T22:24:44.337 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:24:44.337 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:24:44.337 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:24:44.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 = 28 2026-02-25T22:24:44.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:24:44.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:24:44.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2026-02-25T22:24:44.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-25T22:24:44.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:24:44.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:44.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<5\>' 2026-02-25T22:24:44.614 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T22:24:44.614 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:44.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-25T22:24:44.829 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:24:44.829 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:44.829 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:24:44.829 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:24:44.829 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:24:44.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 7 9 4 3 10 2026-02-25T22:24:44.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2026-02-25T22:24:44.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-25T22:24:44.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:24:44.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:44.831 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<1\>' 2026-02-25T22:24:44.831 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T22:24:44.831 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:45.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-25T22:24:45.041 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:24:45.041 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:45.041 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:24:45.041 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:24:45.041 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:24:45.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 7 9 4 3 10 2026-02-25T22:24:45.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:140: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:24:45.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:141: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:24:45.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2026-02-25T22:24:45.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 5 2026-02-25T22:24:45.324 INFO:tasks.workunit.client.0.trial125.stderr:osd.5 is already in. 2026-02-25T22:24:45.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2026-02-25T22:24:45.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 1 2026-02-25T22:24:45.632 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 is already in. 2026-02-25T22:24:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:150: rados_osds_out_in: wait_for_clean 2026-02-25T22:24:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:24:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:24:45.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:24:45.644 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:24:45.644 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:24:45.645 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:24:45.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:24:45.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:24:45.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:24:45.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:24:45.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:24:45.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:24:45.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:24:45.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:24:45.746 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:24:45.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:24:45.958 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:24:45.958 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:24:45.958 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:24:45.958 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:24:45.958 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:24:45.958 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:24:45.958 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:24:45.959 INFO:tasks.workunit.client.0.trial125.stderr:8 2026-02-25T22:24:45.959 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:45.959 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:24:45.959 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:24:45.959 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:45.959 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:24:46.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836523 2026-02-25T22:24:46.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836523 2026-02-25T22:24:46.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836523' 2026-02-25T22:24:46.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:46.037 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:24:46.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673003 2026-02-25T22:24:46.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673003 2026-02-25T22:24:46.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836523 1-42949673003' 2026-02-25T22:24:46.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:46.116 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:24:46.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509482 2026-02-25T22:24:46.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509482 2026-02-25T22:24:46.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509482' 2026-02-25T22:24:46.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:46.195 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:24:46.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743452 2026-02-25T22:24:46.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743452 2026-02-25T22:24:46.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509482 3-296352743452' 2026-02-25T22:24:46.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:46.273 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:24:46.350 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247848 2026-02-25T22:24:46.350 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247848 2026-02-25T22:24:46.350 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509482 3-296352743452 4-98784247848' 2026-02-25T22:24:46.350 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:46.351 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:24:46.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514520 2026-02-25T22:24:46.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514520 2026-02-25T22:24:46.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509482 3-296352743452 4-98784247848 5-326417514520' 2026-02-25T22:24:46.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:46.430 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:24:46.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=373662154773 2026-02-25T22:24:46.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 373662154773 2026-02-25T22:24:46.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509482 3-296352743452 4-98784247848 5-326417514520 6-373662154773' 2026-02-25T22:24:46.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:46.508 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T22:24:46.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=154618822694 2026-02-25T22:24:46.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 154618822694 2026-02-25T22:24:46.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509482 3-296352743452 4-98784247848 5-326417514520 6-373662154773 7-154618822694' 2026-02-25T22:24:46.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:46.587 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T22:24:46.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691877 2026-02-25T22:24:46.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691877 2026-02-25T22:24:46.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509482 3-296352743452 4-98784247848 5-326417514520 6-373662154773 7-154618822694 8-171798691877' 2026-02-25T22:24:46.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:46.665 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T22:24:46.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991249 2026-02-25T22:24:46.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991249 2026-02-25T22:24:46.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509482 3-296352743452 4-98784247848 5-326417514520 6-373662154773 7-154618822694 8-171798691877 9-395136991249' 2026-02-25T22:24:46.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:46.744 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T22:24:46.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397540 2026-02-25T22:24:46.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397540 2026-02-25T22:24:46.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836523 1-42949673003 2-64424509482 3-296352743452 4-98784247848 5-326417514520 6-373662154773 7-154618822694 8-171798691877 9-395136991249 10-210453397540' 2026-02-25T22:24:46.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:46.824 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836523 2026-02-25T22:24:46.825 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:46.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:24:46.826 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836523 2026-02-25T22:24:46.827 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:46.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836523 2026-02-25T22:24:46.828 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 21474836523 2026-02-25T22:24:46.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2026-02-25T22:24:46.828 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:24:47.044 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836523 -lt 21474836523 2026-02-25T22:24:47.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:47.045 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949673003 2026-02-25T22:24:47.046 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:47.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:24:47.048 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673003 2026-02-25T22:24:47.048 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:47.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949673003 2026-02-25T22:24:47.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2026-02-25T22:24:47.050 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 42949673003 2026-02-25T22:24:47.050 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:24:47.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949673003 -lt 42949673003 2026-02-25T22:24:47.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:47.271 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-64424509482 2026-02-25T22:24:47.271 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:47.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:24:47.273 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509482 2026-02-25T22:24:47.273 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:47.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=64424509482 2026-02-25T22:24:47.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2026-02-25T22:24:47.275 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 64424509482 2026-02-25T22:24:47.275 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:24:47.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 64424509482 -lt 64424509482 2026-02-25T22:24:47.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:47.486 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-296352743452 2026-02-25T22:24:47.486 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:47.487 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:24:47.488 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-296352743452 2026-02-25T22:24:47.494 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:47.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=296352743452 2026-02-25T22:24:47.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 296352743452' 2026-02-25T22:24:47.495 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 296352743452 2026-02-25T22:24:47.495 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:24:47.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 296352743452 -lt 296352743452 2026-02-25T22:24:47.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:47.706 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247848 2026-02-25T22:24:47.706 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:47.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:24:47.708 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247848 2026-02-25T22:24:47.708 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:47.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247848 2026-02-25T22:24:47.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247848' 2026-02-25T22:24:47.710 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 98784247848 2026-02-25T22:24:47.710 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:24:47.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247849 -lt 98784247848 2026-02-25T22:24:47.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:47.933 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-326417514520 2026-02-25T22:24:47.933 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:47.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:24:47.935 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-326417514520 2026-02-25T22:24:47.935 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:47.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=326417514520 2026-02-25T22:24:47.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 326417514520' 2026-02-25T22:24:47.937 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 326417514520 2026-02-25T22:24:47.937 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:24:48.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 326417514520 -lt 326417514520 2026-02-25T22:24:48.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:48.155 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-373662154773 2026-02-25T22:24:48.155 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:48.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:24:48.157 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-373662154773 2026-02-25T22:24:48.157 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:48.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=373662154773 2026-02-25T22:24:48.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 373662154773' 2026-02-25T22:24:48.159 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 373662154773 2026-02-25T22:24:48.159 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:24:48.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 373662154773 -lt 373662154773 2026-02-25T22:24:48.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:48.385 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-154618822694 2026-02-25T22:24:48.385 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:48.386 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T22:24:48.387 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-154618822694 2026-02-25T22:24:48.387 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:48.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=154618822694 2026-02-25T22:24:48.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 154618822694' 2026-02-25T22:24:48.388 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.7 seq 154618822694 2026-02-25T22:24:48.389 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T22:24:48.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 154618822694 -lt 154618822694 2026-02-25T22:24:48.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:48.605 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-171798691877 2026-02-25T22:24:48.605 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:48.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T22:24:48.607 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-171798691877 2026-02-25T22:24:48.607 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:48.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691877 2026-02-25T22:24:48.609 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.8 seq 171798691877 2026-02-25T22:24:48.609 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 171798691877' 2026-02-25T22:24:48.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T22:24:48.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691878 -lt 171798691877 2026-02-25T22:24:48.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:48.821 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-395136991249 2026-02-25T22:24:48.821 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:48.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T22:24:48.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-395136991249 2026-02-25T22:24:48.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:48.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=395136991249 2026-02-25T22:24:48.824 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.9 seq 395136991249 2026-02-25T22:24:48.825 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 395136991249' 2026-02-25T22:24:48.825 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T22:24:49.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 395136991249 -lt 395136991249 2026-02-25T22:24:49.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:49.038 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-210453397540 2026-02-25T22:24:49.038 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:49.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T22:24:49.040 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-210453397540 2026-02-25T22:24:49.041 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:49.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397540 2026-02-25T22:24:49.042 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.10 seq 210453397540 2026-02-25T22:24:49.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 210453397540' 2026-02-25T22:24:49.043 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T22:24:49.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397540 -lt 210453397540 2026-02-25T22:24:49.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:24:49.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:24:49.264 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:24:49.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 28 == 0 2026-02-25T22:24:49.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:24:49.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:24:49.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:24:49.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:24:49.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:24:49.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:24:49.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:24:49.748 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=28 2026-02-25T22:24:49.749 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:24:49.749 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:24:49.749 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:24:50.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 = 28 2026-02-25T22:24:50.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:24:50.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:24:50.040 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-25T22:24:50.040 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:24:50.040 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:50.041 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T22:24:50.041 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:50.255 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T22:24:50.256 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:24:50.256 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:50.256 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:24:50.256 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:24:50.256 INFO:tasks.workunit.client.0.trial125.stderr:10' 2026-02-25T22:24:50.256 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2026-02-25T22:24:50.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2026-02-25T22:24:50.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2026-02-25T22:24:50.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:229: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2026-02-25T22:24:50.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-jerasure 2026-02-25T22:24:50.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:24:50.497 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:24:50.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2026-02-25T22:24:50.760 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile profile-jerasure does not exist 2026-02-25T22:24:50.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T22:24:50.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2026-02-25T22:24:50.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2026-02-25T22:24:50.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:157: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2026-02-25T22:24:50.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:158: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2026-02-25T22:24:50.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:160: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2026-02-25T22:24:51.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:165: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2026-02-25T22:24:51.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2026-02-25T22:24:51.420 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-lrc-a' already exists 2026-02-25T22:24:51.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:24:52.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:168: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2026-02-25T22:24:52.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T22:24:52.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2026-02-25T22:24:52.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T22:24:52.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:52.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T22:24:52.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:52.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T22:24:52.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:52.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T22:24:52.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:24:52.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T22:24:52.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T22:24:52.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:24:52.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:24:52.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T22:24:52.482 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2026-02-25T22:24:52.482 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc-a 2026-02-25T22:24:52.482 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:52.482 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2026-02-25T22:24:52.483 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:52.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T22:24:52.697 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:52.697 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:24:52.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 9 3 2026-02-25T22:24:52.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '9' '3') 2026-02-25T22:24:52.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T22:24:52.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2026-02-25T22:24:52.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2026-02-25T22:24:52.941 INFO:tasks.workunit.client.0.trial125.stderr:osd.3 is already out. 2026-02-25T22:24:52.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T22:24:57.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc-a SOMETHING 2026-02-25T22:24:57.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc-a 2026-02-25T22:24:57.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:24:57.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<3\>' 2026-02-25T22:24:57.954 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2026-02-25T22:24:57.954 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:24:58.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T22:24:58.161 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:24:58.161 INFO:tasks.workunit.client.0.trial125.stderr:7' 2026-02-25T22:24:58.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 9 7 2026-02-25T22:24:58.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:24:58.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:24:58.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 3 2026-02-25T22:24:58.411 INFO:tasks.workunit.client.0.trial125.stderr:osd.3 is already in. 2026-02-25T22:24:58.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T22:24:58.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:170: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2026-02-25T22:24:58.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-lrc-a 2026-02-25T22:24:58.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2026-02-25T22:24:58.711 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-lrc-a' does not exist 2026-02-25T22:24:58.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2026-02-25T22:24:58.980 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile profile-lrc-a does not exist 2026-02-25T22:24:58.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T22:24:58.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2026-02-25T22:24:58.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2026-02-25T22:24:58.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:176: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2026-02-25T22:24:58.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:177: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2026-02-25T22:24:58.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2026-02-25T22:24:59.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:183: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2026-02-25T22:24:59.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2026-02-25T22:24:59.627 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-lrc' already exists 2026-02-25T22:24:59.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:25:00.640 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:186: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2026-02-25T22:25:00.640 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T22:25:00.640 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2026-02-25T22:25:00.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T22:25:00.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:00.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T22:25:00.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:00.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T22:25:00.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:00.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T22:25:00.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:00.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T22:25:00.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T22:25:00.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:25:00.689 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:25:00.690 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T22:25:00.692 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2026-02-25T22:25:00.692 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc 2026-02-25T22:25:00.692 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:25:00.693 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc SOMETHING 2026-02-25T22:25:00.693 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:25:00.898 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=8 2026-02-25T22:25:00.898 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:25:00.899 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:25:00.899 INFO:tasks.workunit.client.0.trial125.stderr:10 2026-02-25T22:25:00.899 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:25:00.899 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:25:00.899 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:25:00.899 INFO:tasks.workunit.client.0.trial125.stderr:7' 2026-02-25T22:25:00.899 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 8 6 0 10 5 3 9 7 2026-02-25T22:25:00.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('8' '6' '0' '10' '5' '3' '9' '7') 2026-02-25T22:25:00.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T22:25:00.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2026-02-25T22:25:00.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2026-02-25T22:25:01.149 INFO:tasks.workunit.client.0.trial125.stderr:osd.7 is already out. 2026-02-25T22:25:01.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T22:25:06.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc SOMETHING 2026-02-25T22:25:06.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc 2026-02-25T22:25:06.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:25:06.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2026-02-25T22:25:06.162 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc SOMETHING 2026-02-25T22:25:06.162 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:25:06.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=8 2026-02-25T22:25:06.378 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:25:06.378 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:25:06.378 INFO:tasks.workunit.client.0.trial125.stderr:10 2026-02-25T22:25:06.378 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:25:06.378 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:25:06.378 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:25:06.378 INFO:tasks.workunit.client.0.trial125.stderr:4' 2026-02-25T22:25:06.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 8 6 0 10 5 3 9 4 2026-02-25T22:25:06.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:25:06.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:25:06.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2026-02-25T22:25:06.624 INFO:tasks.workunit.client.0.trial125.stderr:osd.7 is already in. 2026-02-25T22:25:06.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T22:25:06.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:188: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2026-02-25T22:25:06.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-lrc 2026-02-25T22:25:06.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2026-02-25T22:25:06.927 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-lrc' does not exist 2026-02-25T22:25:06.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2026-02-25T22:25:07.194 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile profile-lrc does not exist 2026-02-25T22:25:07.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T22:25:07.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2026-02-25T22:25:07.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:234: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2026-02-25T22:25:07.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:236: TEST_rados_put_get_shec: local poolname=pool-shec 2026-02-25T22:25:07.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:237: TEST_rados_put_get_shec: local profile=profile-shec 2026-02-25T22:25:07.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2026-02-25T22:25:07.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:243: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2026-02-25T22:25:07.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2026-02-25T22:25:07.851 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-shec' already exists 2026-02-25T22:25:07.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:246: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T22:25:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T22:25:08.883 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:25:08.908 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:25:08.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T22:25:08.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2026-02-25T22:25:08.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-shec 2026-02-25T22:25:08.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:25:08.912 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-shec SOMETHING 2026-02-25T22:25:08.912 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:25:09.123 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=7 2026-02-25T22:25:09.123 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:25:09.123 INFO:tasks.workunit.client.0.trial125.stderr:1' 2026-02-25T22:25:09.123 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 7 9 1 2026-02-25T22:25:09.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('7' '9' '1') 2026-02-25T22:25:09.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T22:25:09.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2026-02-25T22:25:09.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2026-02-25T22:25:09.370 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 is already out. 2026-02-25T22:25:09.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T22:25:14.382 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-shec SOMETHING 2026-02-25T22:25:14.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-shec 2026-02-25T22:25:14.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T22:25:14.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<1\>' 2026-02-25T22:25:14.383 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-shec SOMETHING 2026-02-25T22:25:14.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:25:14.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=7 2026-02-25T22:25:14.596 INFO:tasks.workunit.client.0.trial125.stderr:9 2026-02-25T22:25:14.596 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:25:14.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 7 9 6 2026-02-25T22:25:14.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2026-02-25T22:25:14.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T22:25:14.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 1 2026-02-25T22:25:14.843 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 is already in. 2026-02-25T22:25:14.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T22:25:14.854 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:248: TEST_rados_put_get_shec: delete_pool pool-shec 2026-02-25T22:25:14.855 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-shec 2026-02-25T22:25:14.855 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2026-02-25T22:25:15.144 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-shec' does not exist 2026-02-25T22:25:15.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2026-02-25T22:25:15.412 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile profile-shec does not exist 2026-02-25T22:25:15.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2026-02-25T22:25:15.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=ecpool 2026-02-25T22:25:15.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2026-02-25T22:25:15.717 INFO:tasks.workunit.client.0.trial125.stderr:pool 'ecpool' does not exist 2026-02-25T22:25:15.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2026-02-25T22:25:15.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2026-02-25T22:25:15.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:25:15.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2026-02-25T22:25:15.729 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:15.729 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:15.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:15.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:15.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:16.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:16.042 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:16.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:16.044 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:16.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:16.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:16.047 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:16.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:16.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:16.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:16.048 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:16.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:16.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:25:16.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2026-02-25T22:25:16.109 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:16.109 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:16.109 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:25:16.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.45907 2026-02-25T22:25:16.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:16.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:16.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T22:25:16.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/test-erasure-code 0 2026-02-25T22:25:16.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2026-02-25T22:25:16.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T22:25:16.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2026-02-25T22:25:16.112 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:16.112 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:16.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:16.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:16.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:16.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:16.114 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:16.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:16.115 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:16.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:16.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:16.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:16.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:16.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:16.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:16.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:16.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:16.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T22:25:16.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2026-02-25T22:25:16.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:16.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:16.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.45907 2026-02-25T22:25:16.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.45907 2026-02-25T22:25:16.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:16.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:16.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T22:25:16.124 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T22:25:16.125 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T22:25:16.175 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2026-02-25T22:25:16.175 DEBUG:teuthology.orchestra.run.trial125:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61d4431d3ec879bd98659388b3655acf3d392418 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2026-02-25T22:25:16.228 INFO:tasks.workunit.client.0.trial125.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T22:25:16.231 INFO:tasks.workunit.client.0.trial125.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T22:25:16.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:25:16.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:25:16.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T22:25:16.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T22:25:16.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T22:25:16.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T22:25:16.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T22:25:16.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T22:25:16.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/test-erasure-eio 2026-02-25T22:25:16.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2026-02-25T22:25:16.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2026-02-25T22:25:16.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2026-02-25T22:25:16.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2026-02-25T22:25:16.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2026-02-25T22:25:16.233 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2026-02-25T22:25:16.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none ' 2026-02-25T22:25:16.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2026-02-25T22:25:16.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2026-02-25T22:25:16.234 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2026-02-25T22:25:16.234 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T22:25:16.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2026-02-25T22:25:16.236 INFO:tasks.workunit.client.0.trial125.stderr:TEST_ec_object_attr_read_error 2026-02-25T22:25:16.236 INFO:tasks.workunit.client.0.trial125.stderr:TEST_ec_recovery_multiple_errors 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:TEST_ec_recovery_multiple_objects 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:TEST_ec_recovery_multiple_objects_eio 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:TEST_ec_recovery_unfound 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:TEST_ec_single_recovery_error 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_get_bad_size_shard_0 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_get_bad_size_shard_1 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_get_subread_eio_shard_0 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_get_subread_eio_shard_1 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_get_subread_missing 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2026-02-25T22:25:16.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:25:16.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:25:16.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:25:16.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:25:16.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:25:16.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:25:16.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:25:16.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:16.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:16.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:16.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:16.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:16.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:16.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:16.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:16.240 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:16.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:16.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:16.241 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:16.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:16.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:16.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:16.243 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:16.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:16.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:25:16.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:25:16.247 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:16.247 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:16.247 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:16.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:25:16.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:16.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:16.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:25:16.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:25:16.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:16.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:16.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:25:16.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:25:16.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T22:25:16.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T22:25:16.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:25:16.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:25:16.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:25:16.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:25:16.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:25:16.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:25:16.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:25:16.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:25:16.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:25:16.275 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:25:16.275 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:16.275 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:16.276 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:16.276 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:16.276 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:16.276 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:16.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:25:16.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:25:16.313 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:25:16.313 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:16.313 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:25:16.314 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:25:16.315 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:25:16.315 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:25:16.315 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:25:16.315 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:25:16.315 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:16.315 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:16.315 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:16.315 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:25:16.316 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:16.316 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:25:16.370 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:25:16.370 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:16.370 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:25:16.371 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:25:16.371 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:25:16.372 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:25:16.372 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:25:16.372 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:25:16.372 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:16.372 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:16.372 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:16.372 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:25:16.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:16.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:25:16.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:25:16.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:25:16.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:25:16.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:25:16.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:25:16.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:25:16.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:25:16.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:25:16.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:16.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:16.538 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:16.538 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:16.538 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:16.538 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:16.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:25:16.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:25:16.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:25:16.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:25:16.663 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:25:16.671 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:25:17.673 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:25:17.673 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:25:17.673 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:25:17.673 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:17.674 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:17.674 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:17.674 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:25:17.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:25:17.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:25:17.722 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:25:17.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:25:17.729 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:25:16.300+0000 7f855e54cec0 0 load: jerasure load: lrc 2026-02-25T22:25:17.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2026-02-25T22:25:17.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2026-02-25T22:25:17.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: local objname=myobject 2026-02-25T22:25:17.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: local lastobj=300 2026-02-25T22:25:17.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:535: TEST_ec_backfill_unfound: local testobj=obj250 2026-02-25T22:25:17.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:25:17.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:25:17.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: setup_osds 5 2026-02-25T22:25:17.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2026-02-25T22:25:17.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:25:17.732 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2026-02-25T22:25:17.735 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2026-02-25T22:25:17.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:25:17.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:25:17.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:25:17.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:25:17.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:25:17.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:25:17.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:25:17.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:25:17.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:25:17.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:25:17.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:25:17.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:25:17.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:25:17.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:25:17.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:25:17.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:25:17.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:25:17.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:25:17.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:25:17.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:25:17.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:25:17.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:25:17.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:25:17.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:25:17.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:25:17.739 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:25:17.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9a63bb04-e53e-42aa-a1a9-e0e42a7a8c43 2026-02-25T22:25:17.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9a63bb04-e53e-42aa-a1a9-e0e42a7a8c43' 2026-02-25T22:25:17.740 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 9a63bb04-e53e-42aa-a1a9-e0e42a7a8c43 2026-02-25T22:25:17.740 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:25:17.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDNdp9p2RDXLBAA5exkaVjgwbZhqUNf3Ziytg== 2026-02-25T22:25:17.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDNdp9p2RDXLBAA5exkaVjgwbZhqUNf3Ziytg=="}' 2026-02-25T22:25:17.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9a63bb04-e53e-42aa-a1a9-e0e42a7a8c43 -i td/test-erasure-eio/0/new.json 2026-02-25T22:25:17.859 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:25:17.867 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:25:17.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNdp9p2RDXLBAA5exkaVjgwbZhqUNf3Ziytg== --osd-uuid 9a63bb04-e53e-42aa-a1a9-e0e42a7a8c43 2026-02-25T22:25:17.885 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:17.884+0000 7f7bdb8d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:17.887 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:17.888+0000 7f7bdb8d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:17.888 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:17.888+0000 7f7bdb8d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:17.888 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:17.888+0000 7f7bdb8d3a40 -1 bdev(0x556dbdcba400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:25:17.889 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:17.888+0000 7f7bdb8d3a40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:25:18.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:25:18.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:25:18.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:25:18.567 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:25:18.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:25:18.717 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:25:18.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:25:18.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:25:18.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:25:18.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:25:18.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:25:18.733 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:18.732+0000 7f5ca32a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:18.735 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:18.736+0000 7f5ca32a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:18.736 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:18.736+0000 7f5ca32a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:18.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:25:18.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:25:18.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:25:18.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:25:18.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:25:18.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:18.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:25:18.827 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:25:18.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:18.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:25:19.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:19.176 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:19.176+0000 7f5ca32a8a40 -1 Falling back to public interface 2026-02-25T22:25:19.893 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:19.892+0000 7f5ca32a8a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:25:20.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:20.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:20.040 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:25:20.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:25:20.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:20.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:25:20.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:21.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:21.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:21.253 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:25:21.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:25:21.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:21.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:25:21.461 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1926466739,v1:127.0.0.1:6803/1926466739] [v2:127.0.0.1:6804/1926466739,v1:127.0.0.1:6805/1926466739] exists,up 9a63bb04-e53e-42aa-a1a9-e0e42a7a8c43 2026-02-25T22:25:21.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:25:21.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:25:21.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:25:21.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:25:21.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:25:21.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:25:21.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:25:21.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:25:21.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:25:21.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:25:21.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:21.466 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:25:21.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:25:21.468 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:25:21.468 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:25:21.468 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:25:21.468 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:25:21.468 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:25:21.468 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7192ac3f-8606-46ef-8ea4-4429a5e12895 2026-02-25T22:25:21.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7192ac3f-8606-46ef-8ea4-4429a5e12895' 2026-02-25T22:25:21.469 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 7192ac3f-8606-46ef-8ea4-4429a5e12895 2026-02-25T22:25:21.469 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:25:21.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDRdp9pSxrCHBAAWgONxVmMH5Lu6E60T9T7Rw== 2026-02-25T22:25:21.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDRdp9pSxrCHBAAWgONxVmMH5Lu6E60T9T7Rw=="}' 2026-02-25T22:25:21.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7192ac3f-8606-46ef-8ea4-4429a5e12895 -i td/test-erasure-eio/1/new.json 2026-02-25T22:25:21.688 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:25:21.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:25:21.698 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRdp9pSxrCHBAAWgONxVmMH5Lu6E60T9T7Rw== --osd-uuid 7192ac3f-8606-46ef-8ea4-4429a5e12895 2026-02-25T22:25:21.716 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:21.716+0000 7ff871d65a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:21.717 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:21.716+0000 7ff871d65a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:21.718 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:21.720+0000 7ff871d65a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:21.719 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:21.720+0000 7ff871d65a40 -1 bdev(0x563b0fb56400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:25:21.719 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:21.720+0000 7ff871d65a40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:25:22.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:25:22.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:25:22.372 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:25:22.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:25:22.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:25:22.660 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:25:22.661 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:25:22.661 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:25:22.661 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:25:22.661 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:25:22.661 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:25:22.685 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:22.684+0000 7f1392240a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:22.687 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:22.688+0000 7f1392240a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:22.688 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:22.688+0000 7f1392240a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:22.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:25:22.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:25:22.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:25:22.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:25:22.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:25:22.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:22.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:25:22.875 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:25:22.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:22.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:25:23.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:23.092 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:23.091+0000 7f1392240a40 -1 Falling back to public interface 2026-02-25T22:25:23.724 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:23.723+0000 7f1392240a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:25:24.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:24.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:24.089 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:25:24.089 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:25:24.089 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:24.089 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:25:24.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:25.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:25.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:25.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:25:25.300 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:25:25.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:25.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:25:25.509 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1521397130,v1:127.0.0.1:6811/1521397130] [v2:127.0.0.1:6812/1521397130,v1:127.0.0.1:6813/1521397130] exists,up 7192ac3f-8606-46ef-8ea4-4429a5e12895 2026-02-25T22:25:25.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:25:25.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:25:25.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:25:25.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:25:25.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:25:25.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:25:25.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:25:25.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:25:25.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:25:25.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:25:25.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:25:25.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:25:25.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:25:25.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:25:25.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:25:25.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:25:25.516 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:25:25.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=02690b01-2b7c-4213-8e0e-f5c095a56baf 2026-02-25T22:25:25.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 02690b01-2b7c-4213-8e0e-f5c095a56baf' 2026-02-25T22:25:25.516 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 02690b01-2b7c-4213-8e0e-f5c095a56baf 2026-02-25T22:25:25.517 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:25:25.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDVdp9pgieqHxAAGbuMhhovqrKLkObHKvKUSw== 2026-02-25T22:25:25.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDVdp9pgieqHxAAGbuMhhovqrKLkObHKvKUSw=="}' 2026-02-25T22:25:25.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 02690b01-2b7c-4213-8e0e-f5c095a56baf -i td/test-erasure-eio/2/new.json 2026-02-25T22:25:25.746 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:25:25.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:25:25.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVdp9pgieqHxAAGbuMhhovqrKLkObHKvKUSw== --osd-uuid 02690b01-2b7c-4213-8e0e-f5c095a56baf 2026-02-25T22:25:25.782 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:25.779+0000 7f5f5a15fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:25.783 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:25.783+0000 7f5f5a15fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:25.784 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:25.783+0000 7f5f5a15fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:25.785 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:25.783+0000 7f5f5a15fa40 -1 bdev(0x55f0f6b0c400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:25:25.785 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:25.783+0000 7f5f5a15fa40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:25:26.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:25:26.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:25:26.399 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:25:26.400 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:25:26.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:25:26.683 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:25:26.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:25:26.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:25:26.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:25:26.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:25:26.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:25:26.701 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:26.699+0000 7ffb7f7cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:26.703 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:26.703+0000 7ffb7f7cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:26.704 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:26.703+0000 7ffb7f7cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:26.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:25:26.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:25:26.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:25:26.895 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:25:26.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:25:26.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:25:26.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:26.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:25:26.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:26.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:25:27.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:27.148 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:27.147+0000 7ffb7f7cfa40 -1 Falling back to public interface 2026-02-25T22:25:27.804 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:27.803+0000 7ffb7f7cfa40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:25:28.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:28.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:28.111 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:25:28.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:25:28.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:28.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:25:28.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:29.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:29.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:29.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:25:29.333 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:25:29.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:29.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:25:29.537 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3093261780,v1:127.0.0.1:6819/3093261780] [v2:127.0.0.1:6820/3093261780,v1:127.0.0.1:6821/3093261780] exists,up 02690b01-2b7c-4213-8e0e-f5c095a56baf 2026-02-25T22:25:29.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:25:29.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:25:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:25:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:25:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:25:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:25:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:25:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:25:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:25:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:25:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:25:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:25:29.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:29.542 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:25:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:25:29.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:25:29.544 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:25:29.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9e8d277c-e893-47fb-99e5-5547fea994e5 2026-02-25T22:25:29.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 9e8d277c-e893-47fb-99e5-5547fea994e5' 2026-02-25T22:25:29.545 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 9e8d277c-e893-47fb-99e5-5547fea994e5 2026-02-25T22:25:29.545 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:25:29.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDZdp9p4+dNIRAAFkedpxNnXTUxEWsRQNxEKQ== 2026-02-25T22:25:29.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDZdp9p4+dNIRAAFkedpxNnXTUxEWsRQNxEKQ=="}' 2026-02-25T22:25:29.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9e8d277c-e893-47fb-99e5-5547fea994e5 -i td/test-erasure-eio/3/new.json 2026-02-25T22:25:29.762 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:25:29.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:25:29.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZdp9p4+dNIRAAFkedpxNnXTUxEWsRQNxEKQ== --osd-uuid 9e8d277c-e893-47fb-99e5-5547fea994e5 2026-02-25T22:25:29.790 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:29.787+0000 7f47a0f50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:29.791 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:29.791+0000 7f47a0f50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:29.792 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:29.791+0000 7f47a0f50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:29.793 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:29.791+0000 7f47a0f50a40 -1 bdev(0x5592f319c400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:25:29.793 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:29.791+0000 7f47a0f50a40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:25:30.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:25:30.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:25:30.423 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:25:30.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:25:30.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:25:30.699 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:25:30.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:25:30.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:25:30.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:25:30.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:25:30.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:25:30.725 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:30.723+0000 7fb0fd4a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:30.726 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:30.723+0000 7fb0fd4a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:30.727 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:30.727+0000 7fb0fd4a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:30.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:25:30.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:25:30.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:25:30.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:25:30.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:25:30.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:30.912 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:25:30.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:25:30.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:30.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:25:31.104 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:31.103+0000 7fb0fd4a8a40 -1 Falling back to public interface 2026-02-25T22:25:31.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:31.720 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:31.719+0000 7fb0fd4a8a40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:25:32.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:32.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:32.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:25:32.120 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:25:32.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:32.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:25:32.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:33.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:33.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:33.334 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:25:33.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:25:33.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:33.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:25:33.542 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 26 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3654693417,v1:127.0.0.1:6827/3654693417] [v2:127.0.0.1:6828/3654693417,v1:127.0.0.1:6829/3654693417] exists,up 9e8d277c-e893-47fb-99e5-5547fea994e5 2026-02-25T22:25:33.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:25:33.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:25:33.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:25:33.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:25:33.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T22:25:33.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:25:33.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:25:33.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:25:33.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:25:33.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:25:33.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:25:33.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:25:33.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:25:33.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:25:33.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:25:33.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T22:25:33.550 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:25:33.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dbecd710-ca8d-4c9b-93cd-90a7d5930aa3 2026-02-25T22:25:33.550 INFO:tasks.workunit.client.0.trial125.stdout:add osd4 dbecd710-ca8d-4c9b-93cd-90a7d5930aa3 2026-02-25T22:25:33.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 dbecd710-ca8d-4c9b-93cd-90a7d5930aa3' 2026-02-25T22:25:33.551 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:25:33.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDddp9pjdmwIRAAgVRC2724HlAcZmbp/i66zg== 2026-02-25T22:25:33.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDddp9pjdmwIRAAgVRC2724HlAcZmbp/i66zg=="}' 2026-02-25T22:25:33.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dbecd710-ca8d-4c9b-93cd-90a7d5930aa3 -i td/test-erasure-eio/4/new.json 2026-02-25T22:25:33.780 INFO:tasks.workunit.client.0.trial125.stdout:4 2026-02-25T22:25:33.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T22:25:33.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDddp9pjdmwIRAAgVRC2724HlAcZmbp/i66zg== --osd-uuid dbecd710-ca8d-4c9b-93cd-90a7d5930aa3 2026-02-25T22:25:33.807 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:33.807+0000 7fea1e70ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:33.809 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:33.807+0000 7fea1e70ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:33.810 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:33.807+0000 7fea1e70ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:33.811 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:33.807+0000 7fea1e70ea40 -1 bdev(0x55a04bfb6400 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T22:25:33.811 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:33.811+0000 7fea1e70ea40 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T22:25:34.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T22:25:34.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:25:34.454 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T22:25:34.454 INFO:tasks.workunit.client.0.trial125.stdout:adding osd4 key to auth repository 2026-02-25T22:25:34.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:25:34.748 INFO:tasks.workunit.client.0.trial125.stdout:start osd.4 2026-02-25T22:25:34.748 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T22:25:34.748 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:25:34.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:25:34.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:25:34.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:25:34.765 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:34.763+0000 7ff44d65ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:34.767 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:34.767+0000 7ff44d65ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:34.768 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:34.767+0000 7ff44d65ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:34.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T22:25:34.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:25:34.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T22:25:34.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:25:34.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:25:34.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:34.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:25:34.963 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:25:34.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:34.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:25:35.108 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:35.107+0000 7ff44d65ba40 -1 Falling back to public interface 2026-02-25T22:25:35.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:35.724 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:35.723+0000 7ff44d65ba40 -1 osd.4 0 log_to_monitors true 2026-02-25T22:25:36.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:36.177 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:25:36.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:36.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:25:36.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:36.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:25:36.392 INFO:tasks.workunit.client.0.trial125.stdout:osd.4 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2386180636,v1:127.0.0.1:6835/2386180636] [v2:127.0.0.1:6836/2386180636,v1:127.0.0.1:6837/2386180636] exists,up dbecd710-ca8d-4c9b-93cd-90a7d5930aa3 2026-02-25T22:25:36.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:25:36.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:25:36.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:25:36.394 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:25:36.394 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:25:36.394 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:25:36.394 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:36.394 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:36.394 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:36.395 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:25:36.395 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:25:36.395 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:25:36.449 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:25:36.454 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:25:36.457 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:25:19.180+0000 7f5ca32a8a40 0 load: jerasure load: lrc 2026-02-25T22:25:36.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:540: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:25:36.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:542: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2026-02-25T22:25:36.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:543: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T22:25:36.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:25:36.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:25:36.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T22:25:36.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:25:36.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T22:25:36.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:25:36.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T22:25:36.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:25:36.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:25:37.120 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:25:37.130 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:25:38.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:25:38.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:25:38.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:25:38.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:25:38.133 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:25:38.133 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:25:38.133 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:25:38.134 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:25:38.134 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:25:38.134 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:25:38.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:25:38.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:25:38.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:25:38.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:25:38.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:25:38.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:25:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:25:38.451 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:25:38.451 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:25:38.452 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:25:38.452 INFO:tasks.workunit.client.0.trial125.stderr:4' 2026-02-25T22:25:38.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:25:38.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:38.452 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:25:38.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803782 2026-02-25T22:25:38.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803782 2026-02-25T22:25:38.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782' 2026-02-25T22:25:38.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:38.533 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:25:38.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574853 2026-02-25T22:25:38.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574853 2026-02-25T22:25:38.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574853' 2026-02-25T22:25:38.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:38.612 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:25:38.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378628 2026-02-25T22:25:38.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378628 2026-02-25T22:25:38.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574853 2-81604378628' 2026-02-25T22:25:38.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:38.688 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:25:38.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149700 2026-02-25T22:25:38.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149700 2026-02-25T22:25:38.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574853 2-81604378628 3-111669149700' 2026-02-25T22:25:38.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:38.768 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:25:38.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888068 2026-02-25T22:25:38.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888068 2026-02-25T22:25:38.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574853 2-81604378628 3-111669149700 4-146028888068' 2026-02-25T22:25:38.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:38.849 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803782 2026-02-25T22:25:38.850 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:38.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:25:38.851 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803782 2026-02-25T22:25:38.851 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:38.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803782 2026-02-25T22:25:38.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803782' 2026-02-25T22:25:38.853 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803782 2026-02-25T22:25:38.853 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:25:39.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803782 2026-02-25T22:25:39.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:39.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574853 2026-02-25T22:25:39.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:39.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:25:39.069 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574853 2026-02-25T22:25:39.069 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:39.070 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574853 2026-02-25T22:25:39.071 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574853 2026-02-25T22:25:39.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574853' 2026-02-25T22:25:39.071 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:25:39.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574853 -lt 55834574853 2026-02-25T22:25:39.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:39.290 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378628 2026-02-25T22:25:39.290 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:39.291 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:25:39.292 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378628 2026-02-25T22:25:39.292 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:39.294 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378628 2026-02-25T22:25:39.294 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378628' 2026-02-25T22:25:39.294 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 81604378628 2026-02-25T22:25:39.294 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:25:39.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378628 -lt 81604378628 2026-02-25T22:25:39.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:39.502 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149700 2026-02-25T22:25:39.502 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:39.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:25:39.504 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149700 2026-02-25T22:25:39.504 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:39.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149700 2026-02-25T22:25:39.506 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149700 2026-02-25T22:25:39.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149700' 2026-02-25T22:25:39.506 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:25:39.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149699 -lt 111669149700 2026-02-25T22:25:39.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:25:40.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:25:40.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:25:40.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149700 -lt 111669149700 2026-02-25T22:25:40.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:40.937 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-146028888068 2026-02-25T22:25:40.937 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:40.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:25:40.939 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-146028888068 2026-02-25T22:25:40.939 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:40.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=146028888068 2026-02-25T22:25:40.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 146028888068' 2026-02-25T22:25:40.941 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 146028888068 2026-02-25T22:25:40.941 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:25:41.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 146028888068 -lt 146028888068 2026-02-25T22:25:41.154 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:25:41.155 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:25:41.155 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:25:41.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:25:41.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:25:41.442 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:25:41.442 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:25:41.442 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:25:41.442 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:25:41.442 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:25:41.442 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:25:41.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:25:41.651 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:25:41.652 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:25:41.652 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:25:41.929 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:25:41.929 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:25:41.929 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:25:41.929 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-25T22:25:42.127 INFO:tasks.workunit.client.0.trial125.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T22:25:42.127 INFO:tasks.workunit.client.0.trial125.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:37.965202+0000 0'0 42:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-25T22:25:37.071997+0000 0'0 2026-02-25T22:25:37.071997+0000 0 0 periodic scrub scheduled @ 2026-02-27T06:57:14.437050+0000 0 0 2026-02-25T22:25:42.127 INFO:tasks.workunit.client.0.trial125.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:36.946680+0000 0'0 42:23 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T01:02:50.046636+0000 0 0 2026-02-25T22:25:42.127 INFO:tasks.workunit.client.0.trial125.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:36.946981+0000 0'0 42:103 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T08:05:15.983904+0000 0 0 2026-02-25T22:25:42.127 INFO:tasks.workunit.client.0.trial125.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:36.980565+0000 0'0 42:23 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T05:31:31.072459+0000 0 0 2026-02-25T22:25:42.127 INFO:tasks.workunit.client.0.trial125.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:28.539315+0000 0'0 42:65 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T06:29:55.779885+0000 0 0 2026-02-25T22:25:42.127 INFO:tasks.workunit.client.0.trial125.stdout: 2026-02-25T22:25:42.128 INFO:tasks.workunit.client.0.trial125.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T22:25:42.128 INFO:tasks.workunit.client.0.trial125.stderr:dumped pgs 2026-02-25T22:25:42.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-25T22:25:42.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:25:42.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:25:42.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-25T22:25:42.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:42.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:25:42.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:42.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:25:42.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:42.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:25:42.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:25:42.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:25:42.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:42.164 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2026-02-25T22:25:42.164 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=pool-jerasure 2026-02-25T22:25:42.164 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=myobject 2026-02-25T22:25:42.164 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:25:42.164 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-25T22:25:42.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: local primary=3 2026-02-25T22:25:42.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2026-02-25T22:25:42.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:25:42.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T22:25:42.374 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:25:42.375 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:25:42.588 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:25:42.588 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:25:42.588 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:25:42.588 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:25:42.588 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:25:42.588 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2 2026-02-25T22:25:42.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: initial_osds=('3' '1' '4' '0' '2') 2026-02-25T22:25:42.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: local -a initial_osds 2026-02-25T22:25:42.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: local last_osd=2 2026-02-25T22:25:42.589 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T22:25:42.589 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:42.589 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:42.590 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:42.590 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:42.590 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:42.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:42.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: ceph osd down 2 2026-02-25T22:25:43.104 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 is already down. 2026-02-25T22:25:43.113 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: ceph osd out 2 2026-02-25T22:25:43.358 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 is already out. 2026-02-25T22:25:43.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-25T22:25:43.563 INFO:tasks.workunit.client.0.trial125.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T22:25:43.563 INFO:tasks.workunit.client.0.trial125.stdout:2.0 1 0 1 0 0 4096 0 0 1 0 1 active+undersized+degraded 2026-02-25T22:25:42.774709+0000 42'1 44:27 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2026-02-25T22:25:37.071997+0000 0'0 2026-02-25T22:25:37.071997+0000 0 0 periodic scrub scheduled @ 2026-02-27T01:36:29.210469+0000 0 0 2026-02-25T22:25:43.563 INFO:tasks.workunit.client.0.trial125.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:36.946680+0000 0'0 42:23 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T01:02:50.046636+0000 0 0 2026-02-25T22:25:43.563 INFO:tasks.workunit.client.0.trial125.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:36.946981+0000 0'0 42:103 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T08:05:15.983904+0000 0 0 2026-02-25T22:25:43.563 INFO:tasks.workunit.client.0.trial125.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:36.980565+0000 0'0 42:23 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T05:31:31.072459+0000 0 0 2026-02-25T22:25:43.563 INFO:tasks.workunit.client.0.trial125.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:28.539315+0000 0'0 42:65 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T06:29:55.779885+0000 0 0 2026-02-25T22:25:43.563 INFO:tasks.workunit.client.0.trial125.stdout: 2026-02-25T22:25:43.563 INFO:tasks.workunit.client.0.trial125.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T22:25:43.564 INFO:tasks.workunit.client.0.trial125.stderr:dumped pgs 2026-02-25T22:25:43.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2026-02-25T22:25:43.573 INFO:tasks.workunit.client.0.trial125.stderr:4+0 records in 2026-02-25T22:25:43.573 INFO:tasks.workunit.client.0.trial125.stderr:4+0 records out 2026-02-25T22:25:43.573 INFO:tasks.workunit.client.0.trial125.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 8.133e-05 s, 50.4 MB/s 2026-02-25T22:25:43.573 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: seq 1 300 2026-02-25T22:25:43.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.622 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.753 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.806 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.806 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.832 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.832 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.908 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.908 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:43.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:43.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.102 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.102 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.127 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.518 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.518 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.693 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.693 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:44.987 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:44.987 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.070 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.070 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.105 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.105 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.381 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.483 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.483 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:45.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:45.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.189 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.189 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.293 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.293 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.318 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.318 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.386 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.387 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.529 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.529 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.855 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.855 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.907 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.907 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.954 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:46.977 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:46.977 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.126 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.126 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.204 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.204 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.382 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.382 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.785 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.785 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:47.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:47.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.133 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.133 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.328 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.328 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.353 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.353 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.483 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.483 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.637 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.637 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.690 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.690 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.923 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.923 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:48.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:48.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.077 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.077 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.660 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.660 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.780 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.855 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.855 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:49.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:49.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.070 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.197 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.197 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.222 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.222 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.350 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.401 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.454 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.454 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.624 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.624 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.671 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.928 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.929 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:50.980 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:50.980 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:51.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:51.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:51.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:25:51.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2026-02-25T22:25:51.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2026-02-25T22:25:51.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj250 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:25:51.060 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj250 2026-02-25T22:25:51.061 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:25:51.061 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj250 2026-02-25T22:25:51.061 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj250 2026-02-25T22:25:51.061 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:25:51.272 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:25:51.272 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:25:51.272 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:25:51.272 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:25:51.272 INFO:tasks.workunit.client.0.trial125.stderr:2147483647' 2026-02-25T22:25:51.272 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2026-02-25T22:25:51.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-25T22:25:51.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:25:51.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T22:25:51.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:25:51.274 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T22:25:51.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:25:51.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T22:25:51.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:25:51.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T22:25:51.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:25:51.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:25:51.275 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:25:51.276 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T22:25:51.276 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:25:51.276 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:25:51.276 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:51.276 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:51.276 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:51.276 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:25:51.276 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:25:51.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:25:51.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:25:51.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:25:51.337 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T22:25:51.337 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:25:51.337 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:25:51.337 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:51.338 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:51.338 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:51.338 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:25:51.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:25:51.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2026-02-25T22:25:51.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:565: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2026-02-25T22:25:51.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:25:51.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:25:51.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:25:51.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:25:51.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:25:51.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:25:51.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj250 2026-02-25T22:25:51.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:25:51.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:25:51.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:25:51.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T22:25:51.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:25:51.398 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj250 2026-02-25T22:25:51.398 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:25:51.398 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj250 2026-02-25T22:25:51.398 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj250 2026-02-25T22:25:51.398 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:25:51.614 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:25:51.615 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:25:51.615 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:25:51.615 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:25:51.615 INFO:tasks.workunit.client.0.trial125.stderr:2147483647' 2026-02-25T22:25:51.615 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2026-02-25T22:25:51.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-25T22:25:51.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:25:51.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=1 2026-02-25T22:25:51.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:25:51.616 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/1/type 2026-02-25T22:25:51.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:25:51.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-25T22:25:51.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:25:51.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-25T22:25:51.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:25:51.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:25:51.618 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:25:51.618 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-25T22:25:51.618 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:25:51.618 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:25:51.618 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:51.618 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:51.618 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:51.619 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.1.asok 2026-02-25T22:25:51.619 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:25:51.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:25:51.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:25:51.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:25:51.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.1 2026-02-25T22:25:51.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:25:51.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:25:51.680 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:51.680 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:51.680 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:51.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.1.asok 2026-02-25T22:25:51.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:25:51.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2026-02-25T22:25:51.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2026-02-25T22:25:51.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:25:51.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:25:51.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:25:51.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:25:51.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:25:51.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:25:51.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:25:51.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:25:51.742 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:25:51.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T22:25:51.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:25:51.744 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T22:25:51.744 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T22:25:51.744 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:25:51.744 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:25:51.744 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:25:51.759 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:51.762+0000 7fb1b7f59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:51.761 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:51.762+0000 7fb1b7f59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:51.762 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:51.762+0000 7fb1b7f59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:51.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T22:25:51.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:25:51.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:25:51.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:25:51.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:25:51.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:51.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:25:51.947 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:25:51.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:51.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:25:52.137 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:52.138+0000 7fb1b7f59a40 -1 Falling back to public interface 2026-02-25T22:25:52.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:52.791 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:25:52.794+0000 7fb1b7f59a40 -1 osd.2 42 log_to_monitors true 2026-02-25T22:25:53.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:53.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:53.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:25:53.162 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:25:53.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:53.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:25:53.375 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up out weight 0 up_from 48 up_thru 19 down_at 43 last_clean_interval [19,42) [v2:127.0.0.1:6818/1010964852,v1:127.0.0.1:6819/1010964852] [v2:127.0.0.1:6820/1010964852,v1:127.0.0.1:6821/1010964852] exists,up 02690b01-2b7c-4213-8e0e-f5c095a56baf 2026-02-25T22:25:53.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:25:53.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:25:53.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:25:53.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph osd in 2 2026-02-25T22:25:53.624 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 is already in. 2026-02-25T22:25:53.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: sleep 15 2026-02-25T22:26:08.635 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: seq 1 240 2026-02-25T22:26:08.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2026-02-25T22:26:08.637 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2026-02-25T22:26:08.638 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2026-02-25T22:26:08.638 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2026-02-25T22:26:08.638 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2026-02-25T22:26:08.638 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2026-02-25T22:26:08.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2026-02-25T22:26:08.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2026-02-25T22:26:08.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2026-02-25T22:26:08.848 INFO:tasks.workunit.client.0.trial125.stdout:active+backfill_unfound+undersized+degraded+remapped 2026-02-25T22:26:08.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2026-02-25T22:26:08.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: break 2026-02-25T22:26:08.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:582: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-25T22:26:09.044 INFO:tasks.workunit.client.0.trial125.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T22:26:09.044 INFO:tasks.workunit.client.0.trial125.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2026-02-25T22:25:54.097134+0000 46'301 52:899 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2026-02-25T22:25:37.071997+0000 0'0 2026-02-25T22:25:37.071997+0000 0 0 no scrub is scheduled 0 0 2026-02-25T22:26:09.045 INFO:tasks.workunit.client.0.trial125.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:53.672810+0000 0'0 52:60 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-26T23:28:13.282897+0000 0 0 2026-02-25T22:26:09.045 INFO:tasks.workunit.client.0.trial125.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:36.946981+0000 0'0 52:123 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T08:05:15.983904+0000 0 0 2026-02-25T22:26:09.045 INFO:tasks.workunit.client.0.trial125.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:36.980565+0000 0'0 52:42 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T05:31:31.072459+0000 0 0 2026-02-25T22:26:09.045 INFO:tasks.workunit.client.0.trial125.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:53.672998+0000 0'0 52:103 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T07:37:22.050577+0000 0 0 2026-02-25T22:26:09.045 INFO:tasks.workunit.client.0.trial125.stdout: 2026-02-25T22:26:09.045 INFO:tasks.workunit.client.0.trial125.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T22:26:09.045 INFO:tasks.workunit.client.0.trial125.stderr:dumped pgs 2026-02-25T22:26:09.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:583: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T22:26:09.053 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:26:09.053 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:26:09.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:26:09.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:26:09.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:26:09.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:26:09.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: sleep 5 2026-02-25T22:26:14.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:586: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-25T22:26:14.558 INFO:tasks.workunit.client.0.trial125.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T22:26:14.558 INFO:tasks.workunit.client.0.trial125.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2026-02-25T22:25:54.097134+0000 46'301 52:899 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2026-02-25T22:25:37.071997+0000 0'0 2026-02-25T22:25:37.071997+0000 0 0 no scrub is scheduled 0 0 2026-02-25T22:26:14.558 INFO:tasks.workunit.client.0.trial125.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2026-02-25T22:26:09.216222+0000 0'0 54:68 [4,1] 4 [4,1] 4 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T10:03:08.047821+0000 0 0 2026-02-25T22:26:14.558 INFO:tasks.workunit.client.0.trial125.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:36.946981+0000 0'0 54:127 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T08:05:15.983904+0000 0 0 2026-02-25T22:26:14.558 INFO:tasks.workunit.client.0.trial125.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:25:36.980565+0000 0'0 54:46 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T05:31:31.072459+0000 0 0 2026-02-25T22:26:14.558 INFO:tasks.workunit.client.0.trial125.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2026-02-25T22:26:09.216186+0000 0'0 54:111 [1,0] 1 [1,0] 1 0'0 2026-02-25T22:25:16.650211+0000 0'0 2026-02-25T22:25:16.650211+0000 0 0 periodic scrub scheduled @ 2026-02-27T09:52:26.268341+0000 0 0 2026-02-25T22:26:14.558 INFO:tasks.workunit.client.0.trial125.stdout: 2026-02-25T22:26:14.559 INFO:tasks.workunit.client.0.trial125.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T22:26:14.559 INFO:tasks.workunit.client.0.trial125.stderr:dumped pgs 2026-02-25T22:26:14.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-25T22:26:14.640 INFO:tasks.workunit.client.0.trial125.stdout:{ 2026-02-25T22:26:14.640 INFO:tasks.workunit.client.0.trial125.stdout: "num_missing": 1, 2026-02-25T22:26:14.640 INFO:tasks.workunit.client.0.trial125.stdout: "num_unfound": 1, 2026-02-25T22:26:14.640 INFO:tasks.workunit.client.0.trial125.stdout: "objects": [ 2026-02-25T22:26:14.640 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.640 INFO:tasks.workunit.client.0.trial125.stdout: "oid": { 2026-02-25T22:26:14.640 INFO:tasks.workunit.client.0.trial125.stdout: "oid": "obj250", 2026-02-25T22:26:14.640 INFO:tasks.workunit.client.0.trial125.stdout: "key": "", 2026-02-25T22:26:14.640 INFO:tasks.workunit.client.0.trial125.stdout: "snapid": -2, 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: "hash": 2249616407, 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: "max": 0, 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: "pool": 2, 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: "namespace": "" 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: "need": "46'251", 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: "have": "0'0", 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: "flags": "none", 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: "locations": [ 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:26:14.641 INFO:tasks.workunit.client.0.trial125.stdout: ] 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout: "state": "NotRecovering", 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout: "available_might_have_unfound": true, 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout: "might_have_unfound": [ 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout: "osd": "2(4)", 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout: "status": "osd is down" 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout: "more": false 2026-02-25T22:26:14.642 INFO:tasks.workunit.client.0.trial125.stdout:} 2026-02-25T22:26:14.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:588: TEST_ec_backfill_unfound: ceph pg 2.0 query 2026-02-25T22:26:14.732 INFO:tasks.workunit.client.0.trial125.stdout:{ 2026-02-25T22:26:14.733 INFO:tasks.workunit.client.0.trial125.stdout: "snap_trimq": "[]", 2026-02-25T22:26:14.733 INFO:tasks.workunit.client.0.trial125.stdout: "snap_trimq_len": 0, 2026-02-25T22:26:14.733 INFO:tasks.workunit.client.0.trial125.stdout: "state": "active+recovery_unfound+undersized+degraded", 2026-02-25T22:26:14.733 INFO:tasks.workunit.client.0.trial125.stdout: "epoch": 54, 2026-02-25T22:26:14.733 INFO:tasks.workunit.client.0.trial125.stdout: "up": [ 2026-02-25T22:26:14.733 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:26:14.733 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:26:14.733 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:26:14.733 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:26:14.733 INFO:tasks.workunit.client.0.trial125.stdout: 2147483647 2026-02-25T22:26:14.733 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: "acting": [ 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: 2147483647 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: "acting_recovery_backfill": [ 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: "1(1)", 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: "3(0)", 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:26:14.734 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "info": { 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s0", 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "shard": "0", 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "last_update": "46'301", 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "last_complete": "46'250", 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "log_tail": "46'200", 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "last_user_version": 301, 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "last_backfill": "MAX", 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete": [], 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete_epoch": "53", 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [], 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "history": { 2026-02-25T22:26:14.735 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_created": 40, 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_pool_created": 40, 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 54, 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_started": 53, 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_clean": 40, 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_split": 0, 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_marked_full": 0, 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "same_up_since": 53, 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "same_interval_since": 53, 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "same_primary_since": 40, 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:26:14.736 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "prior_readable_until_ub": 0 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "stats": { 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "version": "46'301", 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "reported_seq": 912, 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "reported_epoch": 54, 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "state": "active+recovery_unfound+undersized+degraded", 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "last_fresh": "2026-02-25T22:26:09.218653+0000", 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "last_change": "2026-02-25T22:26:09.218653+0000", 2026-02-25T22:26:14.737 INFO:tasks.workunit.client.0.trial125.stdout: "last_active": "2026-02-25T22:26:09.218653+0000", 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "last_peered": "2026-02-25T22:26:09.218653+0000", 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean": "2026-02-25T22:25:42.164515+0000", 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_active": "2026-02-25T22:26:09.216915+0000", 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_peered": "2026-02-25T22:26:09.216915+0000", 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "last_unstale": "2026-02-25T22:26:09.218653+0000", 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "last_undegraded": "2026-02-25T22:26:09.215585+0000", 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "last_fullsized": "2026-02-25T22:26:09.215428+0000", 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "mapping_epoch": 53, 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "log_start": "46'200", 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_start": "46'200", 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "created": 40, 2026-02-25T22:26:14.738 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "parent": "0.0", 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "parent_split_bits": 0, 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "objects_scrubbed": 0, 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "log_size": 101, 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "log_dups_size": 200, 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_size": 101, 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "stats_invalid": false, 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "dirty_stats_invalid": false, 2026-02-25T22:26:14.739 INFO:tasks.workunit.client.0.trial125.stdout: "omap_stats_invalid": false, 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_stats_invalid": false, 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "pin_stats_invalid": false, 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "manifest_stats_invalid": false, 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrimq_len": 0, 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_duration": 0, 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_schedule": "no scrub is scheduled", 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_duration": 0, 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "objects_trimmed": 0, 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrim_duration": 0, 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "stat_sum": { 2026-02-25T22:26:14.740 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes": 1232896, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects": 301, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_clones": 0, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_copies": 1505, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing_on_primary": 1, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing": 1, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_degraded": 303, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_misplaced": 0, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_unfound": 1, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_dirty": 301, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_whiteouts": 0, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_read": 0, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_read_kb": 0, 2026-02-25T22:26:14.741 INFO:tasks.workunit.client.0.trial125.stdout: "num_write": 301, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_write_kb": 1204, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_scrub_errors": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_deep_scrub_errors": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_recovered": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_recovered": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_keys_recovered": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_omap": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_kb": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict": 0, 2026-02-25T22:26:14.742 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_kb": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_promote": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_high": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_low": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_some": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_full": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_pinned": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_legacy_snapsets": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_large_omap_objects": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_manifest": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_bytes": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_keys": 0, 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_repaired": 0 2026-02-25T22:26:14.743 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: "up": [ 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: 2147483647 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: "acting": [ 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:26:14.744 INFO:tasks.workunit.client.0.trial125.stdout: 2147483647 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: "avail_no_missing": [ 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: "object_location_counts": [ 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: "objects": 300 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: "shards": "0(3),4(2)", 2026-02-25T22:26:14.745 INFO:tasks.workunit.client.0.trial125.stdout: "objects": 1 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: "blocked_by": [], 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: "up_primary": 3, 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: "acting_primary": 3, 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [] 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: "empty": 0, 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: "dne": 0, 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: "incomplete": 0, 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 54, 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: "hit_set_history": { 2026-02-25T22:26:14.746 INFO:tasks.workunit.client.0.trial125.stdout: "current_last_update": "0'0", 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: "history": [] 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: "peer_info": [ 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: "peer": "0(3)", 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s3", 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: "shard": "3", 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: "last_update": "46'301", 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: "last_complete": "46'301", 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: "log_tail": "46'200", 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: "last_user_version": 301, 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: "last_backfill": "MAX", 2026-02-25T22:26:14.747 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete": [], 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [], 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "history": { 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_created": 40, 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_pool_created": 40, 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 54, 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_started": 53, 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_clean": 40, 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_split": 0, 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_marked_full": 0, 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "same_up_since": 53, 2026-02-25T22:26:14.748 INFO:tasks.workunit.client.0.trial125.stdout: "same_interval_since": 53, 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "same_primary_since": 40, 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "prior_readable_until_ub": 0 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "stats": { 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "version": "46'301", 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "reported_seq": 902, 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "reported_epoch": 53, 2026-02-25T22:26:14.749 INFO:tasks.workunit.client.0.trial125.stdout: "state": "active+remapped", 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "last_fresh": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "last_change": "2026-02-25T22:25:42.774709+0000", 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "last_active": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "last_peered": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean": "2026-02-25T22:25:42.164515+0000", 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_active": "2026-02-25T22:25:42.774709+0000", 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_peered": "2026-02-25T22:25:42.774709+0000", 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "last_unstale": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "last_undegraded": "2026-02-25T22:25:42.773525+0000", 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "last_fullsized": "2026-02-25T22:25:42.773413+0000", 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "mapping_epoch": 53, 2026-02-25T22:26:14.750 INFO:tasks.workunit.client.0.trial125.stdout: "log_start": "46'200", 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_start": "46'200", 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "created": 40, 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "parent": "0.0", 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "parent_split_bits": 0, 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "objects_scrubbed": 0, 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "log_size": 101, 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "log_dups_size": 200, 2026-02-25T22:26:14.751 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_size": 101, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "stats_invalid": false, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "dirty_stats_invalid": false, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "omap_stats_invalid": false, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_stats_invalid": false, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "pin_stats_invalid": false, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "manifest_stats_invalid": false, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrimq_len": 0, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_duration": 0, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T01:36:29.210469+0000", 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_duration": 0, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "objects_trimmed": 0, 2026-02-25T22:26:14.752 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrim_duration": 0, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "stat_sum": { 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes": 1232896, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects": 301, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_clones": 0, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_copies": 1505, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing": 0, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_degraded": 301, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_misplaced": 0, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_unfound": 0, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_dirty": 301, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_whiteouts": 0, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_read": 0, 2026-02-25T22:26:14.753 INFO:tasks.workunit.client.0.trial125.stdout: "num_read_kb": 0, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_write": 301, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_write_kb": 1204, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_scrub_errors": 0, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_deep_scrub_errors": 0, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_recovered": 0, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_recovered": 0, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_keys_recovered": 0, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_omap": 0, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush": 0, 2026-02-25T22:26:14.754 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_kb": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_kb": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_promote": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_high": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_low": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_some": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_full": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_pinned": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_legacy_snapsets": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_large_omap_objects": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_manifest": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_bytes": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_keys": 0, 2026-02-25T22:26:14.755 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_repaired": 0 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: "up": [ 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: 2147483647 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: "acting": [ 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:26:14.756 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: 2147483647 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: "avail_no_missing": [ 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: "3(0)", 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: "1(1)", 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: "object_location_counts": [ 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: "objects": 301 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.757 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: "blocked_by": [], 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: "up_primary": 3, 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: "acting_primary": 3, 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [] 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: "empty": 0, 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: "dne": 0, 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: "incomplete": 0, 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 54, 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: "hit_set_history": { 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: "current_last_update": "0'0", 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: "history": [] 2026-02-25T22:26:14.758 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "peer": "1(1)", 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s1", 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "shard": "1", 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "last_update": "46'301", 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "last_complete": "46'301", 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "log_tail": "46'200", 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "last_user_version": 301, 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "last_backfill": "MAX", 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete": [], 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [], 2026-02-25T22:26:14.759 INFO:tasks.workunit.client.0.trial125.stdout: "history": { 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_created": 40, 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_pool_created": 40, 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 54, 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_started": 53, 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_clean": 40, 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_split": 0, 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_marked_full": 0, 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "same_up_since": 53, 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "same_interval_since": 53, 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "same_primary_since": 40, 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:26:14.760 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "prior_readable_until_ub": 0 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "stats": { 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "version": "46'301", 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "reported_seq": 332, 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "reported_epoch": 46, 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "state": "active+remapped", 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "last_fresh": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "last_change": "2026-02-25T22:25:42.774709+0000", 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "last_active": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.761 INFO:tasks.workunit.client.0.trial125.stdout: "last_peered": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean": "2026-02-25T22:25:42.164515+0000", 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_active": "2026-02-25T22:25:42.774709+0000", 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_peered": "2026-02-25T22:25:42.774709+0000", 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "last_unstale": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "last_undegraded": "2026-02-25T22:25:42.773525+0000", 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "last_fullsized": "2026-02-25T22:25:42.773413+0000", 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "mapping_epoch": 53, 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "log_start": "46'200", 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_start": "46'200", 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "created": 40, 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "parent": "0.0", 2026-02-25T22:26:14.762 INFO:tasks.workunit.client.0.trial125.stdout: "parent_split_bits": 0, 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "objects_scrubbed": 0, 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "log_size": 101, 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "log_dups_size": 200, 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_size": 101, 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "stats_invalid": false, 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "dirty_stats_invalid": false, 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "omap_stats_invalid": false, 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_stats_invalid": false, 2026-02-25T22:26:14.763 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "pin_stats_invalid": false, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "manifest_stats_invalid": false, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrimq_len": 0, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_duration": 0, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T01:36:29.210469+0000", 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_duration": 0, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "objects_trimmed": 0, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrim_duration": 0, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "stat_sum": { 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes": 1232896, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects": 301, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_clones": 0, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_copies": 1505, 2026-02-25T22:26:14.764 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing": 1, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_degraded": 301, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_misplaced": 0, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_unfound": 0, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_dirty": 301, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_whiteouts": 0, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_read": 0, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_read_kb": 0, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_write": 301, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_write_kb": 1204, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_scrub_errors": 0, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T22:26:14.765 INFO:tasks.workunit.client.0.trial125.stdout: "num_deep_scrub_errors": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_recovered": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_recovered": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_keys_recovered": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_omap": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_kb": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_kb": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_promote": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_high": 0, 2026-02-25T22:26:14.766 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_low": 0, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_some": 0, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_full": 0, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_pinned": 0, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: "num_legacy_snapsets": 0, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: "num_large_omap_objects": 0, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_manifest": 0, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_bytes": 0, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_keys": 0, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_repaired": 0 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: "up": [ 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:26:14.767 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: 2147483647 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: "acting": [ 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: 2147483647 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: "avail_no_missing": [ 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: "3(0)", 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:26:14.768 INFO:tasks.workunit.client.0.trial125.stdout: "1(1)", 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: "object_location_counts": [ 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: "objects": 301 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: "blocked_by": [], 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: "up_primary": 3, 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: "acting_primary": 3, 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [] 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: "empty": 0, 2026-02-25T22:26:14.769 INFO:tasks.workunit.client.0.trial125.stdout: "dne": 0, 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "incomplete": 0, 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 54, 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "hit_set_history": { 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "current_last_update": "0'0", 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "history": [] 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "peer": "4(2)", 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s2", 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "shard": "2", 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "last_update": "46'301", 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "last_complete": "46'301", 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "log_tail": "46'200", 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "last_user_version": 301, 2026-02-25T22:26:14.770 INFO:tasks.workunit.client.0.trial125.stdout: "last_backfill": "MAX", 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete": [], 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [], 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "history": { 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_created": 40, 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_pool_created": 40, 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 54, 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_started": 53, 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_clean": 40, 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_split": 0, 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_marked_full": 0, 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "same_up_since": 53, 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "same_interval_since": 53, 2026-02-25T22:26:14.771 INFO:tasks.workunit.client.0.trial125.stdout: "same_primary_since": 40, 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "prior_readable_until_ub": 0 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "stats": { 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "version": "46'301", 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "reported_seq": 332, 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "reported_epoch": 46, 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "state": "active+remapped", 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "last_fresh": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "last_change": "2026-02-25T22:25:42.774709+0000", 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "last_active": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "last_peered": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.772 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean": "2026-02-25T22:25:42.164515+0000", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_active": "2026-02-25T22:25:42.774709+0000", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_peered": "2026-02-25T22:25:42.774709+0000", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "last_unstale": "2026-02-25T22:25:51.059941+0000", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "last_undegraded": "2026-02-25T22:25:42.773525+0000", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "last_fullsized": "2026-02-25T22:25:42.773413+0000", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "mapping_epoch": 53, 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "log_start": "46'200", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_start": "46'200", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "created": 40, 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "parent": "0.0", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "parent_split_bits": 0, 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.773 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:25:37.071997+0000", 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "objects_scrubbed": 0, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "log_size": 101, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "log_dups_size": 200, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_size": 101, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "stats_invalid": false, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "dirty_stats_invalid": false, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "omap_stats_invalid": false, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_stats_invalid": false, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "pin_stats_invalid": false, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "manifest_stats_invalid": false, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrimq_len": 0, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_duration": 0, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T01:36:29.210469+0000", 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_duration": 0, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "objects_trimmed": 0, 2026-02-25T22:26:14.774 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrim_duration": 0, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "stat_sum": { 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes": 1232896, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects": 301, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_clones": 0, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_copies": 1505, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing": 0, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_degraded": 301, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_misplaced": 0, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_unfound": 0, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_dirty": 301, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_whiteouts": 0, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_read": 0, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_read_kb": 0, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_write": 301, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_write_kb": 1204, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_scrub_errors": 0, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T22:26:14.775 INFO:tasks.workunit.client.0.trial125.stdout: "num_deep_scrub_errors": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_recovered": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_recovered": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_keys_recovered": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_omap": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_kb": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_kb": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_promote": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_high": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_low": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_some": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_full": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_pinned": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_legacy_snapsets": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_large_omap_objects": 0, 2026-02-25T22:26:14.776 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_manifest": 0, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_bytes": 0, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_keys": 0, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_repaired": 0 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: "up": [ 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: 2147483647 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: "acting": [ 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: 2147483647 2026-02-25T22:26:14.777 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "avail_no_missing": [ 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "3(0)", 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "1(1)", 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "object_location_counts": [ 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "objects": 301 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "blocked_by": [], 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "up_primary": 3, 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "acting_primary": 3, 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [] 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "empty": 0, 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "dne": 0, 2026-02-25T22:26:14.778 INFO:tasks.workunit.client.0.trial125.stdout: "incomplete": 0, 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 54, 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "hit_set_history": { 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "current_last_update": "0'0", 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "history": [] 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "recovery_state": [ 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "name": "Started/Primary/Active", 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "enter_time": "2026-02-25T22:26:09.215444+0000", 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "might_have_unfound": [ 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "osd": "0(3)", 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "status": "already probed" 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "osd": "1(1)", 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: "status": "already probed" 2026-02-25T22:26:14.779 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "osd": "2(4)", 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "status": "osd is down" 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "osd": "4(2)", 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "status": "already probed" 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "recovery_progress": { 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "backfill_targets": [], 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "waiting_on_backfill": [], 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_backfill_started": "MIN", 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "backfill_info": { 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "begin": "MIN", 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "end": "MIN", 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "objects": [] 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "peer_backfill_info": [], 2026-02-25T22:26:14.780 INFO:tasks.workunit.client.0.trial125.stdout: "backfills_in_flight": [], 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "recovering": [], 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "pg_backend": { 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "recovery_ops": [], 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "read_ops": [] 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "name": "Started", 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "enter_time": "2026-02-25T22:26:09.115283+0000" 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "scrubber": { 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "active": false, 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "must_scrub": false, 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "must_deep_scrub": false, 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "must_repair": false, 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "sched_time": "2026-02-27T01:36:29.210469+0000", 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "schedule": "not registered for scrubbing", 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "urgency": "periodic-regular", 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "shallow-target": { 2026-02-25T22:26:14.781 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s0", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "level": "shallow", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "urgency": "periodic-regular", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "sched_time": "2026-02-27T01:36:29.210469+0000", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "orig_sched_time": "2026-02-27T01:36:29.210469+0000", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "last_issue": "ok", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "forced": false, 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "eligible": false, 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "queued": false, 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "active": false 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "deep-target": { 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s0", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "level": "deep", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "urgency": "periodic-regular", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "sched_time": "2026-03-04T07:36:34.103960+0000", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "orig_sched_time": "2026-03-04T07:36:34.103960+0000", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "last_issue": "ok", 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "forced": false, 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "eligible": false, 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "queued": false, 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: "active": false 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:26:14.782 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:26:14.783 INFO:tasks.workunit.client.0.trial125.stdout: "agent_state": {} 2026-02-25T22:26:14.783 INFO:tasks.workunit.client.0.trial125.stdout:} 2026-02-25T22:26:14.783 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:590: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-25T22:26:14.783 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:590: TEST_ec_backfill_unfound: grep -q obj250 2026-02-25T22:26:14.820 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-25T22:26:14.820 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2026-02-25T22:26:14.902 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: check=true 2026-02-25T22:26:14.902 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: test true == true 2026-02-25T22:26:14.902 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-25T22:26:14.903 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2026-02-25T22:26:14.980 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2026-02-25T22:26:14.980 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: check='osd is down' 2026-02-25T22:26:14.980 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:596: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2026-02-25T22:26:14.981 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-25T22:26:14.981 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2026-02-25T22:26:15.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2026-02-25T22:26:15.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: check='2(4)' 2026-02-25T22:26:15.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:599: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2026-02-25T22:26:15.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:601: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2026-02-25T22:26:15.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:26:15.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:26:15.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T22:26:15.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:26:15.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:26:15.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:26:15.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:26:15.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:26:15.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:26:15.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:26:15.065 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:26:15.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T22:26:15.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:26:15.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T22:26:15.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T22:26:15.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:26:15.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:26:15.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:26:15.081 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:15.082+0000 7f3a80288a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:15.083 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:15.082+0000 7f3a80288a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:15.083 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:15.086+0000 7f3a80288a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:15.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T22:26:15.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:26:15.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:26:15.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:26:15.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:26:15.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:15.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:26:15.275 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:26:15.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:15.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:26:15.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:15.504 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:15.506+0000 7f3a80288a40 -1 Falling back to public interface 2026-02-25T22:26:16.165 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:16.166+0000 7f3a80288a40 -1 osd.2 52 log_to_monitors true 2026-02-25T22:26:16.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:16.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:16.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:26:16.502 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:26:16.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:16.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:26:16.708 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 56 up_thru 19 down_at 53 last_clean_interval [48,52) [v2:127.0.0.1:6818/1495597200,v1:127.0.0.1:6819/1495597200] [v2:127.0.0.1:6820/1495597200,v1:127.0.0.1:6821/1495597200] exists,up 02690b01-2b7c-4213-8e0e-f5c095a56baf 2026-02-25T22:26:16.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:26:16.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:26:16.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:26:16.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2026-02-25T22:26:21.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:605: TEST_ec_backfill_unfound: test 124 = 124 2026-02-25T22:26:21.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:607: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2026-02-25T22:26:21.785 INFO:tasks.workunit.client.0.trial125.stderr:pg has no unfound objects 2026-02-25T22:26:21.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_backfill_unfound: wait_for_clean 2026-02-25T22:26:21.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:26:21.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:26:21.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:26:21.794 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:26:21.794 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:26:21.794 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:26:21.794 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:26:21.794 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:26:21.794 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:26:21.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:26:21.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:26:21.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:26:21.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:26:21.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:26:21.906 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:26:22.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:26:22.108 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:26:22.108 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:26:22.108 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:26:22.108 INFO:tasks.workunit.client.0.trial125.stderr:4' 2026-02-25T22:26:22.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:26:22.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:26:22.108 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:26:22.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803793 2026-02-25T22:26:22.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803793 2026-02-25T22:26:22.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803793' 2026-02-25T22:26:22.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:26:22.186 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:26:22.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574864 2026-02-25T22:26:22.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574864 2026-02-25T22:26:22.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803793 1-55834574864' 2026-02-25T22:26:22.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:26:22.262 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:26:22.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=240518168580 2026-02-25T22:26:22.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 240518168580 2026-02-25T22:26:22.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803793 1-55834574864 2-240518168580' 2026-02-25T22:26:22.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:26:22.341 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:26:22.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149710 2026-02-25T22:26:22.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149710 2026-02-25T22:26:22.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803793 1-55834574864 2-240518168580 3-111669149710' 2026-02-25T22:26:22.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:26:22.417 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:26:22.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888079 2026-02-25T22:26:22.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888079 2026-02-25T22:26:22.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803793 1-55834574864 2-240518168580 3-111669149710 4-146028888079' 2026-02-25T22:26:22.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:26:22.498 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803793 2026-02-25T22:26:22.498 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:26:22.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:26:22.499 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803793 2026-02-25T22:26:22.500 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:26:22.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803793 2026-02-25T22:26:22.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803793' 2026-02-25T22:26:22.501 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803793 2026-02-25T22:26:22.502 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:26:22.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803790 -lt 25769803793 2026-02-25T22:26:22.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:26:23.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:26:23.717 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:26:23.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803793 -lt 25769803793 2026-02-25T22:26:23.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:26:23.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574864 2026-02-25T22:26:23.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:26:23.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:26:23.934 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574864 2026-02-25T22:26:23.935 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:26:23.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574864 2026-02-25T22:26:23.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574864' 2026-02-25T22:26:23.936 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574864 2026-02-25T22:26:23.937 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:26:24.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574864 -lt 55834574864 2026-02-25T22:26:24.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:26:24.158 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-240518168580 2026-02-25T22:26:24.158 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:26:24.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:26:24.160 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-240518168580 2026-02-25T22:26:24.160 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:26:24.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=240518168580 2026-02-25T22:26:24.161 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 240518168580 2026-02-25T22:26:24.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 240518168580' 2026-02-25T22:26:24.162 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:26:24.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 240518168580 -lt 240518168580 2026-02-25T22:26:24.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:26:24.378 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149710 2026-02-25T22:26:24.379 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:26:24.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:26:24.380 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149710 2026-02-25T22:26:24.380 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:26:24.382 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149710 2026-02-25T22:26:24.382 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149710 2026-02-25T22:26:24.382 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149710' 2026-02-25T22:26:24.383 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:26:24.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149710 -lt 111669149710 2026-02-25T22:26:24.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:26:24.607 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-146028888079 2026-02-25T22:26:24.607 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:26:24.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:26:24.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-146028888079 2026-02-25T22:26:24.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:26:24.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=146028888079 2026-02-25T22:26:24.611 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 146028888079 2026-02-25T22:26:24.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 146028888079' 2026-02-25T22:26:24.611 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:26:24.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 146028888079 -lt 146028888079 2026-02-25T22:26:24.834 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:26:24.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:26:24.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:26:25.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:26:25.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:26:25.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:26:25.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:26:25.118 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:26:25.118 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:26:25.118 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:26:25.118 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:26:25.323 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:26:25.324 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:26:25.324 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:26:25.324 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:26:25.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:26:25.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:26:25.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:26:25.614 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: seq 1 300 2026-02-25T22:26:25.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2026-02-25T22:26:25.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2026-02-25T22:26:25.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2026-02-25T22:26:25.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2026-02-25T22:26:25.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.702 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2026-02-25T22:26:25.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2026-02-25T22:26:25.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2026-02-25T22:26:25.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.766 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2026-02-25T22:26:25.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.786 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.787 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.787 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2026-02-25T22:26:25.787 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2026-02-25T22:26:25.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.833 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2026-02-25T22:26:25.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.858 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.858 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2026-02-25T22:26:25.859 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2026-02-25T22:26:25.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2026-02-25T22:26:25.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2026-02-25T22:26:25.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2026-02-25T22:26:25.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.966 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2026-02-25T22:26:25.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2026-02-25T22:26:25.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:25.991 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:25.991 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2026-02-25T22:26:25.991 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2026-02-25T22:26:26.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2026-02-25T22:26:26.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2026-02-25T22:26:26.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2026-02-25T22:26:26.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.106 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2026-02-25T22:26:26.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.130 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2026-02-25T22:26:26.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2026-02-25T22:26:26.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2026-02-25T22:26:26.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2026-02-25T22:26:26.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2026-02-25T22:26:26.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2026-02-25T22:26:26.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2026-02-25T22:26:26.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2026-02-25T22:26:26.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2026-02-25T22:26:26.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.328 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.328 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2026-02-25T22:26:26.328 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2026-02-25T22:26:26.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2026-02-25T22:26:26.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2026-02-25T22:26:26.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2026-02-25T22:26:26.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2026-02-25T22:26:26.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.454 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2026-02-25T22:26:26.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2026-02-25T22:26:26.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2026-02-25T22:26:26.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2026-02-25T22:26:26.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2026-02-25T22:26:26.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2026-02-25T22:26:26.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.594 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2026-02-25T22:26:26.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2026-02-25T22:26:26.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2026-02-25T22:26:26.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2026-02-25T22:26:26.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.689 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2026-02-25T22:26:26.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2026-02-25T22:26:26.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2026-02-25T22:26:26.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.757 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2026-02-25T22:26:26.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2026-02-25T22:26:26.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2026-02-25T22:26:26.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2026-02-25T22:26:26.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.854 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.854 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2026-02-25T22:26:26.854 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2026-02-25T22:26:26.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2026-02-25T22:26:26.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2026-02-25T22:26:26.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2026-02-25T22:26:26.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.959 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2026-02-25T22:26:26.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.979 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.980 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2026-02-25T22:26:26.980 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2026-02-25T22:26:26.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:26.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:26.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2026-02-25T22:26:26.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2026-02-25T22:26:27.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2026-02-25T22:26:27.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2026-02-25T22:26:27.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2026-02-25T22:26:27.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.106 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2026-02-25T22:26:27.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.127 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2026-02-25T22:26:27.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2026-02-25T22:26:27.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2026-02-25T22:26:27.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2026-02-25T22:26:27.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2026-02-25T22:26:27.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2026-02-25T22:26:27.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2026-02-25T22:26:27.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2026-02-25T22:26:27.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2026-02-25T22:26:27.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.349 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.349 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2026-02-25T22:26:27.349 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2026-02-25T22:26:27.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.399 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.399 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2026-02-25T22:26:27.399 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2026-02-25T22:26:27.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2026-02-25T22:26:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.468 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2026-02-25T22:26:27.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.491 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.492 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2026-02-25T22:26:27.492 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2026-02-25T22:26:27.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2026-02-25T22:26:27.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2026-02-25T22:26:27.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2026-02-25T22:26:27.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.603 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.603 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2026-02-25T22:26:27.603 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2026-02-25T22:26:27.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2026-02-25T22:26:27.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.675 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.675 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2026-02-25T22:26:27.675 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.698 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2026-02-25T22:26:27.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2026-02-25T22:26:27.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.750 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.751 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.751 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2026-02-25T22:26:27.751 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2026-02-25T22:26:27.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2026-02-25T22:26:27.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2026-02-25T22:26:27.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2026-02-25T22:26:27.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2026-02-25T22:26:27.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.889 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2026-02-25T22:26:27.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.908 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2026-02-25T22:26:27.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.929 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2026-02-25T22:26:27.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2026-02-25T22:26:27.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2026-02-25T22:26:27.980 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:27.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:27.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2026-02-25T22:26:27.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2026-02-25T22:26:28.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2026-02-25T22:26:28.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2026-02-25T22:26:28.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2026-02-25T22:26:28.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.098 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2026-02-25T22:26:28.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2026-02-25T22:26:28.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2026-02-25T22:26:28.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2026-02-25T22:26:28.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2026-02-25T22:26:28.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2026-02-25T22:26:28.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.222 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2026-02-25T22:26:28.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2026-02-25T22:26:28.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2026-02-25T22:26:28.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2026-02-25T22:26:28.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2026-02-25T22:26:28.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2026-02-25T22:26:28.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2026-02-25T22:26:28.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.386 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.386 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2026-02-25T22:26:28.386 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2026-02-25T22:26:28.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2026-02-25T22:26:28.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2026-02-25T22:26:28.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.482 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.483 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.483 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2026-02-25T22:26:28.483 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2026-02-25T22:26:28.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2026-02-25T22:26:28.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2026-02-25T22:26:28.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2026-02-25T22:26:28.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2026-02-25T22:26:28.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2026-02-25T22:26:28.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2026-02-25T22:26:28.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2026-02-25T22:26:28.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2026-02-25T22:26:28.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2026-02-25T22:26:28.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.748 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2026-02-25T22:26:28.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2026-02-25T22:26:28.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.788 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2026-02-25T22:26:28.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.810 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2026-02-25T22:26:28.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.829 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2026-02-25T22:26:28.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2026-02-25T22:26:28.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2026-02-25T22:26:28.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2026-02-25T22:26:28.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2026-02-25T22:26:28.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2026-02-25T22:26:28.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2026-02-25T22:26:28.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2026-02-25T22:26:28.986 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:28.987 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:28.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2026-02-25T22:26:28.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2026-02-25T22:26:29.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2026-02-25T22:26:29.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2026-02-25T22:26:29.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2026-02-25T22:26:29.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2026-02-25T22:26:29.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2026-02-25T22:26:29.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2026-02-25T22:26:29.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2026-02-25T22:26:29.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2026-02-25T22:26:29.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.200 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2026-02-25T22:26:29.200 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.218 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2026-02-25T22:26:29.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2026-02-25T22:26:29.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2026-02-25T22:26:29.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2026-02-25T22:26:29.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2026-02-25T22:26:29.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2026-02-25T22:26:29.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.354 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2026-02-25T22:26:29.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.379 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.379 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2026-02-25T22:26:29.379 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2026-02-25T22:26:29.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2026-02-25T22:26:29.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2026-02-25T22:26:29.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2026-02-25T22:26:29.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2026-02-25T22:26:29.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2026-02-25T22:26:29.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2026-02-25T22:26:29.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2026-02-25T22:26:29.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2026-02-25T22:26:29.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2026-02-25T22:26:29.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.631 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.632 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2026-02-25T22:26:29.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2026-02-25T22:26:29.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2026-02-25T22:26:29.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.702 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.702 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2026-02-25T22:26:29.702 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2026-02-25T22:26:29.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2026-02-25T22:26:29.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2026-02-25T22:26:29.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2026-02-25T22:26:29.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2026-02-25T22:26:29.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2026-02-25T22:26:29.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.867 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2026-02-25T22:26:29.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2026-02-25T22:26:29.893 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2026-02-25T22:26:29.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2026-02-25T22:26:29.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.966 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.966 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2026-02-25T22:26:29.966 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2026-02-25T22:26:29.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:29.991 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:29.991 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2026-02-25T22:26:29.991 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2026-02-25T22:26:30.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2026-02-25T22:26:30.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2026-02-25T22:26:30.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2026-02-25T22:26:30.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2026-02-25T22:26:30.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.133 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2026-02-25T22:26:30.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2026-02-25T22:26:30.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2026-02-25T22:26:30.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2026-02-25T22:26:30.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2026-02-25T22:26:30.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2026-02-25T22:26:30.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2026-02-25T22:26:30.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2026-02-25T22:26:30.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.323 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2026-02-25T22:26:30.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.346 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2026-02-25T22:26:30.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2026-02-25T22:26:30.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2026-02-25T22:26:30.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.414 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2026-02-25T22:26:30.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.439 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.439 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2026-02-25T22:26:30.440 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2026-02-25T22:26:30.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2026-02-25T22:26:30.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2026-02-25T22:26:30.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2026-02-25T22:26:30.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2026-02-25T22:26:30.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.570 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.571 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.571 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2026-02-25T22:26:30.571 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.594 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2026-02-25T22:26:30.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2026-02-25T22:26:30.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.637 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2026-02-25T22:26:30.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.661 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2026-02-25T22:26:30.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2026-02-25T22:26:30.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2026-02-25T22:26:30.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2026-02-25T22:26:30.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.748 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.748 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2026-02-25T22:26:30.748 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2026-02-25T22:26:30.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2026-02-25T22:26:30.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2026-02-25T22:26:30.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2026-02-25T22:26:30.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2026-02-25T22:26:30.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2026-02-25T22:26:30.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2026-02-25T22:26:30.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2026-02-25T22:26:30.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2026-02-25T22:26:30.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2026-02-25T22:26:30.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2026-02-25T22:26:30.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:30.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:30.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2026-02-25T22:26:30.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2026-02-25T22:26:31.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.034 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2026-02-25T22:26:31.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2026-02-25T22:26:31.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.078 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.079 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.080 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2026-02-25T22:26:31.080 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2026-02-25T22:26:31.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2026-02-25T22:26:31.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2026-02-25T22:26:31.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2026-02-25T22:26:31.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2026-02-25T22:26:31.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2026-02-25T22:26:31.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2026-02-25T22:26:31.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2026-02-25T22:26:31.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2026-02-25T22:26:31.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.300 INFO:tasks.workunit.client.0.trial125.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2026-02-25T22:26:31.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2026-02-25T22:26:31.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.328 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.328 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2026-02-25T22:26:31.328 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2026-02-25T22:26:31.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2026-02-25T22:26:31.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2026-02-25T22:26:31.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.413 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2026-02-25T22:26:31.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.440 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.440 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2026-02-25T22:26:31.440 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2026-02-25T22:26:31.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2026-02-25T22:26:31.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2026-02-25T22:26:31.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2026-02-25T22:26:31.529 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2026-02-25T22:26:31.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2026-02-25T22:26:31.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2026-02-25T22:26:31.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2026-02-25T22:26:31.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2026-02-25T22:26:31.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.648 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.649 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.649 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2026-02-25T22:26:31.649 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2026-02-25T22:26:31.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.698 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2026-02-25T22:26:31.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2026-02-25T22:26:31.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2026-02-25T22:26:31.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2026-02-25T22:26:31.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2026-02-25T22:26:31.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2026-02-25T22:26:31.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2026-02-25T22:26:31.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.860 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2026-02-25T22:26:31.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2026-02-25T22:26:31.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2026-02-25T22:26:31.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.922 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.923 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.924 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2026-02-25T22:26:31.924 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2026-02-25T22:26:31.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2026-02-25T22:26:31.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2026-02-25T22:26:31.995 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:31.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:31.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2026-02-25T22:26:31.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2026-02-25T22:26:32.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2026-02-25T22:26:32.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.070 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2026-02-25T22:26:32.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2026-02-25T22:26:32.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2026-02-25T22:26:32.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.144 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2026-02-25T22:26:32.144 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2026-02-25T22:26:32.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2026-02-25T22:26:32.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2026-02-25T22:26:32.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2026-02-25T22:26:32.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2026-02-25T22:26:32.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2026-02-25T22:26:32.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2026-02-25T22:26:32.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2026-02-25T22:26:32.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.354 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2026-02-25T22:26:32.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2026-02-25T22:26:32.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.395 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2026-02-25T22:26:32.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:26:32.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2026-02-25T22:26:32.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2026-02-25T22:26:32.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.439 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:26:32.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:26:32.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:26:32.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:26:32.698 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:26:32.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:26:33.004 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:26:33.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:26:33.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:26:33.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:26:33.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:26:33.017 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:26:33.017 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:26:33.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:26:33.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:26:33.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:26:33.127 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:26:33.128 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:26:33.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:26:33.129 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:26:33.130 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:26:33.130 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:26:33.130 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:26:33.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:33.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:26:33.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:26:33.132 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:33.133 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:26:33.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:26:33.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:26:33.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:26:33.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:33.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:33.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:26:33.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:26:33.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:26:33.184 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:26:33.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:26:33.185 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:26:33.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:26:33.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:26:33.186 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:26:33.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:33.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:26:33.188 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:26:33.188 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:33.189 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:26:33.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:26:33.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:26:33.192 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:26:33.192 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:33.192 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:33.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:26:33.193 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:26:33.193 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:26:33.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:26:33.195 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:26:33.195 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:33.195 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:33.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:26:33.197 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:26:33.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:26:33.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:26:33.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:26:33.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:26:33.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:26:33.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:26:33.198 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:26:33.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:26:33.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:26:33.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:26:33.220 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:26:33.220 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:33.220 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:33.221 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:33.221 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:33.221 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:33.221 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:33.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:26:33.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:26:33.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:26:33.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:33.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:26:33.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:26:33.253 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:26:33.253 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:26:33.253 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:26:33.254 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:26:33.254 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:33.254 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:33.254 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:33.254 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:26:33.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:33.255 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:26:33.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:26:33.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:33.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:26:33.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:26:33.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:26:33.310 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:26:33.310 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:26:33.311 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:26:33.311 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:33.311 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:33.311 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:33.311 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:26:33.311 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:33.311 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:26:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:26:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:26:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:26:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:26:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:26:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:26:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:26:33.474 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:26:33.475 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:33.475 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:33.475 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:33.475 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:33.476 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:33.476 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:33.476 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:26:33.476 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:26:33.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:26:33.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:26:33.613 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:26:33.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:26:34.623 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:26:34.624 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:26:34.624 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:26:34.624 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:34.624 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:34.624 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:34.624 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:26:34.624 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:26:34.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:26:34.672 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:26:34.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:26:34.680 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:26:33.241+0000 7fe1af7aaec0 0 load: jerasure load: lrc 2026-02-25T22:26:34.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2026-02-25T22:26:34.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:379: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2026-02-25T22:26:34.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:380: TEST_ec_object_attr_read_error: local objname=myobject 2026-02-25T22:26:34.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:382: TEST_ec_object_attr_read_error: setup_osds 7 2026-02-25T22:26:34.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-25T22:26:34.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:26:34.681 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-25T22:26:34.682 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:26:34.684 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:26:34.685 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:26:34.686 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d2861c86-8de5-4a70-a1f5-a0b4d83b32d1 2026-02-25T22:26:34.686 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 d2861c86-8de5-4a70-a1f5-a0b4d83b32d1 2026-02-25T22:26:34.686 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d2861c86-8de5-4a70-a1f5-a0b4d83b32d1' 2026-02-25T22:26:34.686 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:26:34.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAad59ps5W1KRAAsuT7vMmPcFQuSWZ5WMI2Hw== 2026-02-25T22:26:34.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAad59ps5W1KRAAsuT7vMmPcFQuSWZ5WMI2Hw=="}' 2026-02-25T22:26:34.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d2861c86-8de5-4a70-a1f5-a0b4d83b32d1 -i td/test-erasure-eio/0/new.json 2026-02-25T22:26:34.808 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:26:34.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:26:34.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAad59ps5W1KRAAsuT7vMmPcFQuSWZ5WMI2Hw== --osd-uuid d2861c86-8de5-4a70-a1f5-a0b4d83b32d1 2026-02-25T22:26:34.833 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:34.833+0000 7ff4d0397a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:34.834 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:34.833+0000 7ff4d0397a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:34.835 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:34.833+0000 7ff4d0397a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:34.836 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:34.833+0000 7ff4d0397a40 -1 bdev(0x557242a06400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:26:34.836 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:34.833+0000 7ff4d0397a40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:26:35.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:26:35.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:26:35.564 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:26:35.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:26:35.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:26:35.698 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:26:35.698 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:26:35.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:26:35.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:26:35.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:26:35.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:26:35.714 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:35.712+0000 7f5b93c98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:35.716 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:35.712+0000 7f5b93c98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:35.716 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:35.716+0000 7f5b93c98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:35.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:26:35.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:26:35.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:26:35.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:26:35.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:26:35.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:35.805 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:26:35.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:26:35.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:35.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:26:36.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:36.190 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:36.188+0000 7f5b93c98a40 -1 Falling back to public interface 2026-02-25T22:26:36.861 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:36.860+0000 7f5b93c98a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:26:37.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:37.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:37.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:26:37.017 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:26:37.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:37.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:26:37.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:38.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:38.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:38.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:26:38.237 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:26:38.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:38.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:26:38.449 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2429354416,v1:127.0.0.1:6803/2429354416] [v2:127.0.0.1:6804/2429354416,v1:127.0.0.1:6805/2429354416] exists,up d2861c86-8de5-4a70-a1f5-a0b4d83b32d1 2026-02-25T22:26:38.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:26:38.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:26:38.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:26:38.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:38.451 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:38.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:38.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:26:38.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:26:38.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:26:38.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:26:38.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e60713bc-7f6b-4588-8753-355829c50491 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 e60713bc-7f6b-4588-8753-355829c50491 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e60713bc-7f6b-4588-8753-355829c50491' 2026-02-25T22:26:38.453 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:26:38.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAed59pKzTcGxAALjLY5cFMVlVPwl4KDPVAVg== 2026-02-25T22:26:38.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAed59pKzTcGxAALjLY5cFMVlVPwl4KDPVAVg=="}' 2026-02-25T22:26:38.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e60713bc-7f6b-4588-8753-355829c50491 -i td/test-erasure-eio/1/new.json 2026-02-25T22:26:38.690 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:26:38.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:26:38.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAed59pKzTcGxAALjLY5cFMVlVPwl4KDPVAVg== --osd-uuid e60713bc-7f6b-4588-8753-355829c50491 2026-02-25T22:26:38.717 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:38.716+0000 7f6229124a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:38.719 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:38.716+0000 7f6229124a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:38.720 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:38.716+0000 7f6229124a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:38.720 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:38.716+0000 7f6229124a40 -1 bdev(0x5597ab2fa400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:26:38.720 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:38.716+0000 7f6229124a40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:26:39.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:26:39.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:26:39.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:26:39.309 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:26:39.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:26:39.586 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:26:39.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:26:39.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:26:39.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:26:39.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:26:39.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:26:39.609 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:39.608+0000 7f505ee1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:39.611 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:39.608+0000 7f505ee1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:39.612 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:39.608+0000 7f505ee1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:39.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:26:39.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:26:39.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:26:39.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:26:39.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:26:39.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:39.796 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:26:39.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:26:39.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:39.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:26:39.958 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:39.956+0000 7f505ee1ba40 -1 Falling back to public interface 2026-02-25T22:26:40.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:40.622 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:40.620+0000 7f505ee1ba40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:26:41.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:41.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:41.011 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:26:41.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:26:41.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:41.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:26:41.227 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1284396883,v1:127.0.0.1:6811/1284396883] [v2:127.0.0.1:6812/1284396883,v1:127.0.0.1:6813/1284396883] exists,up e60713bc-7f6b-4588-8753-355829c50491 2026-02-25T22:26:41.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:26:41.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:26:41.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:26:41.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:26:41.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:26:41.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:26:41.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:26:41.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:26:41.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:26:41.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:26:41.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:26:41.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:26:41.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:26:41.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:26:41.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:26:41.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:26:41.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:26:41.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:26:41.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:26:41.237 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:26:41.237 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:41.237 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:41.237 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:41.237 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:41.237 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:41.237 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:41.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:41.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:26:41.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:26:41.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:26:41.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:26:41.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:26:41.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:26:41.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:26:41.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:26:41.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:26:41.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:26:41.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:26:41.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:26:41.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:26:41.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6eef4d18-2ee1-45d6-8fca-397b3b237a24 2026-02-25T22:26:41.240 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 6eef4d18-2ee1-45d6-8fca-397b3b237a24 2026-02-25T22:26:41.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6eef4d18-2ee1-45d6-8fca-397b3b237a24' 2026-02-25T22:26:41.241 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:26:41.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAhd59pBrmXDhAAb7UkpnyXbWOs4N1jzd7P5Q== 2026-02-25T22:26:41.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAhd59pBrmXDhAAb7UkpnyXbWOs4N1jzd7P5Q=="}' 2026-02-25T22:26:41.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6eef4d18-2ee1-45d6-8fca-397b3b237a24 -i td/test-erasure-eio/2/new.json 2026-02-25T22:26:41.468 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:26:41.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:26:41.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhd59pBrmXDhAAb7UkpnyXbWOs4N1jzd7P5Q== --osd-uuid 6eef4d18-2ee1-45d6-8fca-397b3b237a24 2026-02-25T22:26:41.495 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:41.492+0000 7f165402fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:41.496 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:41.492+0000 7f165402fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:41.497 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:41.492+0000 7f165402fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:41.498 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:41.496+0000 7f165402fa40 -1 bdev(0x557343f5a400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:26:41.498 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:41.496+0000 7f165402fa40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:26:42.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:26:42.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:26:42.130 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:26:42.130 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:26:42.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:26:42.431 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:26:42.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:26:42.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:26:42.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:26:42.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:26:42.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:26:42.456 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:42.452+0000 7f3ab45b8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:42.457 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:42.456+0000 7f3ab45b8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:42.458 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:42.456+0000 7f3ab45b8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:42.637 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:26:42.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:26:42.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:26:42.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:26:42.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:26:42.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:42.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:26:42.639 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:26:42.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:42.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:26:42.807 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:42.804+0000 7f3ab45b8a40 -1 Falling back to public interface 2026-02-25T22:26:42.854 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:43.486 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:43.484+0000 7f3ab45b8a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:26:43.855 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:43.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:43.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:26:43.856 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:26:43.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:43.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:26:44.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:45.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:45.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:45.071 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:26:45.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:26:45.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:45.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:26:45.280 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2515775106,v1:127.0.0.1:6819/2515775106] [v2:127.0.0.1:6820/2515775106,v1:127.0.0.1:6821/2515775106] exists,up 6eef4d18-2ee1-45d6-8fca-397b3b237a24 2026-02-25T22:26:45.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:26:45.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:26:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:26:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:26:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:26:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:26:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:26:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:26:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:26:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:26:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:26:45.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:45.285 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:26:45.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:26:45.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:26:45.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:26:45.287 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:26:45.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bbc64da7-4ec0-4196-b9bd-1468fc8c60cd 2026-02-25T22:26:45.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 bbc64da7-4ec0-4196-b9bd-1468fc8c60cd' 2026-02-25T22:26:45.288 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 bbc64da7-4ec0-4196-b9bd-1468fc8c60cd 2026-02-25T22:26:45.288 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:26:45.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAld59pSry4ERAAt09GjRRgX0TvxEJpolIOlg== 2026-02-25T22:26:45.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAld59pSry4ERAAt09GjRRgX0TvxEJpolIOlg=="}' 2026-02-25T22:26:45.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bbc64da7-4ec0-4196-b9bd-1468fc8c60cd -i td/test-erasure-eio/3/new.json 2026-02-25T22:26:45.520 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:26:45.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:26:45.531 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAld59pSry4ERAAt09GjRRgX0TvxEJpolIOlg== --osd-uuid bbc64da7-4ec0-4196-b9bd-1468fc8c60cd 2026-02-25T22:26:45.546 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:45.543+0000 7fda8bc62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:45.548 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:45.543+0000 7fda8bc62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:45.549 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:45.543+0000 7fda8bc62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:45.550 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:45.547+0000 7fda8bc62a40 -1 bdev(0x55c924ef8400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:26:45.550 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:45.547+0000 7fda8bc62a40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:26:46.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:26:46.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:26:46.181 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:26:46.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:26:46.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:26:46.473 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:26:46.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:26:46.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:26:46.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:26:46.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:26:46.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:26:46.497 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:46.491+0000 7f0dd82b8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:46.499 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:46.495+0000 7f0dd82b8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:46.500 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:46.495+0000 7f0dd82b8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:46.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:26:46.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:26:46.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:26:46.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:26:46.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:26:46.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:46.681 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:26:46.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:26:46.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:46.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:26:46.843 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:46.839+0000 7f0dd82b8a40 -1 Falling back to public interface 2026-02-25T22:26:46.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:47.440 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:47.435+0000 7f0dd82b8a40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:26:47.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:47.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:47.894 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:26:47.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:26:47.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:47.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:26:48.106 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2494747800,v1:127.0.0.1:6827/2494747800] [v2:127.0.0.1:6828/2494747800,v1:127.0.0.1:6829/2494747800] exists,up bbc64da7-4ec0-4196-b9bd-1468fc8c60cd 2026-02-25T22:26:48.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:26:48.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:26:48.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:26:48.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:48.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:26:48.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T22:26:48.113 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:26:48.113 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8263c64e-7b33-4de7-94a8-adc73daf758e 2026-02-25T22:26:48.113 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 8263c64e-7b33-4de7-94a8-adc73daf758e' 2026-02-25T22:26:48.114 INFO:tasks.workunit.client.0.trial125.stdout:add osd4 8263c64e-7b33-4de7-94a8-adc73daf758e 2026-02-25T22:26:48.114 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:26:48.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAod59pE91sBxAAb7V1+62pCoscRiHlwcSUQw== 2026-02-25T22:26:48.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAod59pE91sBxAAb7V1+62pCoscRiHlwcSUQw=="}' 2026-02-25T22:26:48.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8263c64e-7b33-4de7-94a8-adc73daf758e -i td/test-erasure-eio/4/new.json 2026-02-25T22:26:48.342 INFO:tasks.workunit.client.0.trial125.stdout:4 2026-02-25T22:26:48.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T22:26:48.353 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAod59pE91sBxAAb7V1+62pCoscRiHlwcSUQw== --osd-uuid 8263c64e-7b33-4de7-94a8-adc73daf758e 2026-02-25T22:26:48.369 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:48.363+0000 7f01ecb4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:48.371 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:48.367+0000 7f01ecb4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:48.372 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:48.367+0000 7f01ecb4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:48.372 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:48.367+0000 7f01ecb4ea40 -1 bdev(0x562d8856c400 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T22:26:48.372 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:48.367+0000 7f01ecb4ea40 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T22:26:49.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T22:26:49.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:26:49.065 INFO:tasks.workunit.client.0.trial125.stdout:adding osd4 key to auth repository 2026-02-25T22:26:49.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T22:26:49.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:26:49.364 INFO:tasks.workunit.client.0.trial125.stdout:start osd.4 2026-02-25T22:26:49.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T22:26:49.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:26:49.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:26:49.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:26:49.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:26:49.389 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:49.383+0000 7ff21f7b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:49.390 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:49.387+0000 7ff21f7b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:49.391 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:49.387+0000 7ff21f7b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:49.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T22:26:49.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:26:49.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T22:26:49.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:26:49.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:26:49.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:49.575 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:26:49.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:26:49.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:49.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:26:49.785 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:49.823 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:49.819+0000 7ff21f7b6a40 -1 Falling back to public interface 2026-02-25T22:26:50.544 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:50.539+0000 7ff21f7b6a40 -1 osd.4 0 log_to_monitors true 2026-02-25T22:26:50.786 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:50.787 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:50.787 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:26:50.788 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:26:50.788 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:50.788 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:26:50.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:51.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:51.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:51.999 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:26:52.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:26:52.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:52.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:26:52.215 INFO:tasks.workunit.client.0.trial125.stdout:osd.4 up in weight 1 up_from 35 up_thru 36 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4115111487,v1:127.0.0.1:6835/4115111487] [v2:127.0.0.1:6836/4115111487,v1:127.0.0.1:6837/4115111487] exists,up 8263c64e-7b33-4de7-94a8-adc73daf758e 2026-02-25T22:26:52.218 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:26:52.218 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:26:52.218 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:26:52.218 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:26:52.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-25T22:26:52.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:26:52.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:26:52.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T22:26:52.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:26:52.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-25T22:26:52.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:26:52.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:26:52.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:26:52.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:26:52.219 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:52.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:26:52.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:26:52.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:26:52.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:26:52.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:26:52.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:26:52.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:26:52.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:26:52.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:26:52.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:26:52.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:26:52.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-25T22:26:52.222 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:26:52.222 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=43c6650c-f971-4946-9a38-dd2a56e65698 2026-02-25T22:26:52.222 INFO:tasks.workunit.client.0.trial125.stdout:add osd5 43c6650c-f971-4946-9a38-dd2a56e65698 2026-02-25T22:26:52.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 43c6650c-f971-4946-9a38-dd2a56e65698' 2026-02-25T22:26:52.223 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:26:52.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAsd59pZEfTDRAAye/dCOor73ThiY6uNjlanA== 2026-02-25T22:26:52.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAsd59pZEfTDRAAye/dCOor73ThiY6uNjlanA=="}' 2026-02-25T22:26:52.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 43c6650c-f971-4946-9a38-dd2a56e65698 -i td/test-erasure-eio/5/new.json 2026-02-25T22:26:52.438 INFO:tasks.workunit.client.0.trial125.stdout:5 2026-02-25T22:26:52.449 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-25T22:26:52.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsd59pZEfTDRAAye/dCOor73ThiY6uNjlanA== --osd-uuid 43c6650c-f971-4946-9a38-dd2a56e65698 2026-02-25T22:26:52.472 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:52.467+0000 7fc3abecba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:52.474 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:52.471+0000 7fc3abecba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:52.475 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:52.471+0000 7fc3abecba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:52.476 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:52.471+0000 7fc3abecba40 -1 bdev(0x561ac7d92400 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-25T22:26:52.476 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:52.471+0000 7fc3abecba40 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-25T22:26:53.113 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-25T22:26:53.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:26:53.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T22:26:53.115 INFO:tasks.workunit.client.0.trial125.stdout:adding osd5 key to auth repository 2026-02-25T22:26:53.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:26:53.402 INFO:tasks.workunit.client.0.trial125.stdout:start osd.5 2026-02-25T22:26:53.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T22:26:53.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:26:53.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:26:53.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:26:53.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:26:53.418 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:53.419+0000 7f524bc42a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:53.420 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:53.419+0000 7f524bc42a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:53.420 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:53.419+0000 7f524bc42a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:53.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T22:26:53.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:26:53.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T22:26:53.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:26:53.609 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:26:53.609 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:53.609 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:26:53.609 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:26:53.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:53.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:26:53.811 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:53.811+0000 7f524bc42a40 -1 Falling back to public interface 2026-02-25T22:26:53.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:54.375 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:54.375+0000 7f524bc42a40 -1 osd.5 0 log_to_monitors true 2026-02-25T22:26:54.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:54.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:54.822 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:26:54.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:26:54.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:54.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:26:55.034 INFO:tasks.workunit.client.0.trial125.stdout:osd.5 up in weight 1 up_from 43 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3063474001,v1:127.0.0.1:6843/3063474001] [v2:127.0.0.1:6844/3063474001,v1:127.0.0.1:6845/3063474001] exists,up 43c6650c-f971-4946-9a38-dd2a56e65698 2026-02-25T22:26:55.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:26:55.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:55.039 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:26:55.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:26:55.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-25T22:26:55.041 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:26:55.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4043babc-7412-49a3-a1d6-c425e9955228 2026-02-25T22:26:55.042 INFO:tasks.workunit.client.0.trial125.stdout:add osd6 4043babc-7412-49a3-a1d6-c425e9955228 2026-02-25T22:26:55.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 4043babc-7412-49a3-a1d6-c425e9955228' 2026-02-25T22:26:55.042 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:26:55.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAvd59pV5EbAxAAssEXEkiUVme90B43xjhs6A== 2026-02-25T22:26:55.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAvd59pV5EbAxAAssEXEkiUVme90B43xjhs6A=="}' 2026-02-25T22:26:55.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4043babc-7412-49a3-a1d6-c425e9955228 -i td/test-erasure-eio/6/new.json 2026-02-25T22:26:55.272 INFO:tasks.workunit.client.0.trial125.stdout:6 2026-02-25T22:26:55.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-25T22:26:55.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvd59pV5EbAxAAssEXEkiUVme90B43xjhs6A== --osd-uuid 4043babc-7412-49a3-a1d6-c425e9955228 2026-02-25T22:26:55.298 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:55.299+0000 7f5295f44a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:55.300 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:55.299+0000 7f5295f44a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:55.301 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:55.299+0000 7f5295f44a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:55.301 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:55.299+0000 7f5295f44a40 -1 bdev(0x55b253702400 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-25T22:26:55.301 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:55.299+0000 7f5295f44a40 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-25T22:26:55.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-25T22:26:55.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:26:55.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T22:26:55.946 INFO:tasks.workunit.client.0.trial125.stdout:adding osd6 key to auth repository 2026-02-25T22:26:55.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:26:56.240 INFO:tasks.workunit.client.0.trial125.stdout:start osd.6 2026-02-25T22:26:56.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T22:26:56.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:26:56.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:26:56.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:26:56.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:26:56.264 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:56.262+0000 7fe4be28aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:56.266 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:56.266+0000 7fe4be28aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:56.267 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:56.266+0000 7fe4be28aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:26:56.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T22:26:56.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:26:56.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T22:26:56.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:26:56.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:26:56.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:56.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:26:56.452 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:26:56.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:56.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:26:56.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:56.683 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:56.682+0000 7fe4be28aa40 -1 Falling back to public interface 2026-02-25T22:26:57.271 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:26:57.270+0000 7fe4be28aa40 -1 osd.6 0 log_to_monitors true 2026-02-25T22:26:57.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:57.668 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:26:57.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:57.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:26:57.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:57.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:26:57.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:26:58.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:26:58.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:26:58.880 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:26:58.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:26:58.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:26:58.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:26:59.093 INFO:tasks.workunit.client.0.trial125.stdout:osd.6 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2703386975,v1:127.0.0.1:6851/2703386975] [v2:127.0.0.1:6852/2703386975,v1:127.0.0.1:6853/2703386975] exists,up 4043babc-7412-49a3-a1d6-c425e9955228 2026-02-25T22:26:59.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:26:59.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:26:59.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:26:59.094 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:26:59.094 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:26:59.094 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:26:59.095 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:59.095 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:59.095 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:26:59.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:26:59.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:26:59.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:26:59.159 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:26:59.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:26:59.168 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:26:36.192+0000 7f5b93c98a40 0 load: jerasure load: lrc 2026-02-25T22:26:59.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:384: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2026-02-25T22:26:59.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T22:26:59.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:26:59.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:26:59.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T22:26:59.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:26:59.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T22:26:59.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:26:59.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T22:26:59.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:26:59.471 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:26:59.839 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:26:59.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:27:00.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:27:00.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:27:00.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:27:00.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:27:00.851 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:27:00.852 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:27:00.852 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:27:00.853 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:27:00.853 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:27:00.853 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:27:00.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:27:00.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:27:00.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:27:00.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:27:00.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:27:00.956 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:27:01.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:27:01.161 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:27:01.161 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:27:01.161 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:27:01.161 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:27:01.161 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:27:01.161 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:27:01.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:27:01.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:01.162 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:27:01.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803783 2026-02-25T22:27:01.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803783 2026-02-25T22:27:01.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783' 2026-02-25T22:27:01.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:01.240 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:27:01.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574854 2026-02-25T22:27:01.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574854 2026-02-25T22:27:01.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574854' 2026-02-25T22:27:01.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:01.319 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:27:01.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411334 2026-02-25T22:27:01.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411334 2026-02-25T22:27:01.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574854 2-77309411334' 2026-02-25T22:27:01.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:01.396 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:27:01.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116997 2026-02-25T22:27:01.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116997 2026-02-25T22:27:01.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574854 2-77309411334 3-115964116997' 2026-02-25T22:27:01.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:01.476 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:27:01.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855364 2026-02-25T22:27:01.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855364 2026-02-25T22:27:01.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574854 2-77309411334 3-115964116997 4-150323855364' 2026-02-25T22:27:01.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:01.557 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:27:01.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593732 2026-02-25T22:27:01.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593732 2026-02-25T22:27:01.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574854 2-77309411334 3-115964116997 4-150323855364 5-184683593732' 2026-02-25T22:27:01.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:01.635 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:27:01.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=223338299395 2026-02-25T22:27:01.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 223338299395 2026-02-25T22:27:01.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574854 2-77309411334 3-115964116997 4-150323855364 5-184683593732 6-223338299395' 2026-02-25T22:27:01.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:01.713 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803783 2026-02-25T22:27:01.713 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:01.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:27:01.715 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803783 2026-02-25T22:27:01.715 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:01.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803783 2026-02-25T22:27:01.716 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803783 2026-02-25T22:27:01.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803783' 2026-02-25T22:27:01.717 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:27:01.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803783 -lt 25769803783 2026-02-25T22:27:01.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:01.938 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574854 2026-02-25T22:27:01.938 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:01.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:27:01.940 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574854 2026-02-25T22:27:01.940 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:01.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574854 2026-02-25T22:27:01.941 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574854 2026-02-25T22:27:01.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574854' 2026-02-25T22:27:01.942 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:27:02.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574854 -lt 55834574854 2026-02-25T22:27:02.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:02.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411334 2026-02-25T22:27:02.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:02.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:27:02.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411334 2026-02-25T22:27:02.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:02.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411334 2026-02-25T22:27:02.166 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 77309411334 2026-02-25T22:27:02.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411334' 2026-02-25T22:27:02.167 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:27:02.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411334 -lt 77309411334 2026-02-25T22:27:02.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:02.381 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116997 2026-02-25T22:27:02.381 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:02.382 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:27:02.383 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116997 2026-02-25T22:27:02.383 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:02.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116997 2026-02-25T22:27:02.384 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 115964116997 2026-02-25T22:27:02.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116997' 2026-02-25T22:27:02.385 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:27:02.589 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116997 -lt 115964116997 2026-02-25T22:27:02.589 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:02.590 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855364 2026-02-25T22:27:02.590 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:02.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:27:02.592 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855364 2026-02-25T22:27:02.592 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:02.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855364 2026-02-25T22:27:02.593 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 150323855364 2026-02-25T22:27:02.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855364' 2026-02-25T22:27:02.594 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:27:02.810 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855365 -lt 150323855364 2026-02-25T22:27:02.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:02.811 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-184683593732 2026-02-25T22:27:02.811 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:02.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:27:02.813 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-184683593732 2026-02-25T22:27:02.813 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:02.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593732 2026-02-25T22:27:02.815 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 184683593732 2026-02-25T22:27:02.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 184683593732' 2026-02-25T22:27:02.815 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:27:03.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593732 -lt 184683593732 2026-02-25T22:27:03.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:03.030 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-223338299395 2026-02-25T22:27:03.030 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:03.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:27:03.032 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-223338299395 2026-02-25T22:27:03.032 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:03.033 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=223338299395 2026-02-25T22:27:03.033 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 223338299395 2026-02-25T22:27:03.034 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 223338299395' 2026-02-25T22:27:03.034 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:27:03.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 223338299395 -lt 223338299395 2026-02-25T22:27:03.253 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:27:03.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:27:03.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:27:03.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:27:03.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:27:03.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:27:03.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:27:03.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:27:03.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:27:03.529 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:27:03.529 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:27:03.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:27:03.739 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:27:03.740 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:27:03.740 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:27:04.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:27:04.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:27:04.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:27:04.037 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:387: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2026-02-25T22:27:04.037 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=pool-jerasure 2026-02-25T22:27:04.037 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=myobject 2026-02-25T22:27:04.037 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:27:04.037 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-25T22:27:04.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:387: TEST_ec_object_attr_read_error: local primary_osd=3 2026-02-25T22:27:04.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:389: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-25T22:27:04.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:27:04.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:27:04.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:27:04.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:27:04.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:392: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:27:04.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:27:04.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-25T22:27:04.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2026-02-25T22:27:04.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:27:04.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:27:04.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=mdata 2026-02-25T22:27:04.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:27:04.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:27:04.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:27:04.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T22:27:04.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:27:04.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:27:04.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:27:04.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T22:27:04.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:27:04.935 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T22:27:04.935 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:27:04.935 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T22:27:04.936 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:27:04.936 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:27:05.149 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2147483647 2026-02-25T22:27:05.149 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:27:05.149 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:27:05.149 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:27:05.149 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:27:05.150 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2147483647 5 1 0 6 2026-02-25T22:27:05.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('2147483647' '5' '1' '0' '6') 2026-02-25T22:27:05.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:27:05.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=5 2026-02-25T22:27:05.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:27:05.150 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/5/type 2026-02-25T22:27:05.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:27:05.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2026-02-25T22:27:05.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:27:05.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=5 2026-02-25T22:27:05.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:27:05.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:27:05.152 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:27:05.153 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.5 2026-02-25T22:27:05.153 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.5 2026-02-25T22:27:05.153 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.5 ']' 2026-02-25T22:27:05.153 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:05.153 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:05.153 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:05.153 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.5.asok 2026-02-25T22:27:05.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:27:05.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:27:05.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:27:05.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:27:05.214 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.5 2026-02-25T22:27:05.214 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.5 2026-02-25T22:27:05.214 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.5 ']' 2026-02-25T22:27:05.215 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:05.215 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:05.215 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:05.215 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.5.asok 2026-02-25T22:27:05.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:27:05.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2026-02-25T22:27:05.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2026-02-25T22:27:05.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:27:05.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:27:05.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T22:27:05.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:27:05.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:27:05.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:27:05.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:27:05.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:27:05.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:27:05.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:27:05.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:27:05.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:27:05.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:27:05.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:27:05.277 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:27:05.277 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:05.277 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:05.278 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:05.278 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:05.278 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:27:05.279 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:27:05.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T22:27:05.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:27:05.280 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-25T22:27:05.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T22:27:05.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:27:05.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:27:05.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:27:05.298 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:05.294+0000 7f16d9399a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:05.300 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:05.298+0000 7f16d9399a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:05.301 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:05.298+0000 7f16d9399a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:05.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T22:27:05.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:27:05.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:27:05.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:27:05.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:27:05.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:05.491 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:27:05.491 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:27:05.491 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:05.491 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:27:05.680 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:05.678+0000 7f16d9399a40 -1 Falling back to public interface 2026-02-25T22:27:05.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:06.330 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:06.330+0000 7f16d9399a40 -1 osd.3 59 log_to_monitors true 2026-02-25T22:27:06.702 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:06.702 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:06.702 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:27:06.702 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:27:06.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:06.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:27:06.908 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 63 up_thru 63 down_at 60 last_clean_interval [27,59) [v2:127.0.0.1:6826/1070268915,v1:127.0.0.1:6827/1070268915] [v2:127.0.0.1:6828/1070268915,v1:127.0.0.1:6829/1070268915] exists,up bbc64da7-4ec0-4196-b9bd-1468fc8c60cd 2026-02-25T22:27:06.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:27:06.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:27:06.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:27:06.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_object_attr_read_error: wait_for_clean 2026-02-25T22:27:06.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:27:06.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:27:06.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:27:06.909 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:27:06.910 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:27:06.910 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:27:06.910 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:27:06.910 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:27:06.910 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:27:07.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:27:07.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:27:07.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:27:07.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:27:07.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:27:07.008 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:27:07.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:27:07.221 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:27:07.221 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:27:07.221 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:27:07.221 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:27:07.221 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:27:07.221 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:27:07.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:27:07.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:07.221 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:27:07.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803786 2026-02-25T22:27:07.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803786 2026-02-25T22:27:07.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786' 2026-02-25T22:27:07.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:07.300 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:27:07.379 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574858 2026-02-25T22:27:07.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574858 2026-02-25T22:27:07.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-55834574858' 2026-02-25T22:27:07.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:07.380 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:27:07.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411337 2026-02-25T22:27:07.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411337 2026-02-25T22:27:07.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-55834574858 2-77309411337' 2026-02-25T22:27:07.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:07.460 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:27:07.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=270582939651 2026-02-25T22:27:07.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 270582939651 2026-02-25T22:27:07.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-55834574858 2-77309411337 3-270582939651' 2026-02-25T22:27:07.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:07.537 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:27:07.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855368 2026-02-25T22:27:07.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855368 2026-02-25T22:27:07.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-55834574858 2-77309411337 3-270582939651 4-150323855368' 2026-02-25T22:27:07.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:07.617 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:27:07.695 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593735 2026-02-25T22:27:07.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593735 2026-02-25T22:27:07.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-55834574858 2-77309411337 3-270582939651 4-150323855368 5-184683593735' 2026-02-25T22:27:07.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:07.696 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:27:07.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=223338299398 2026-02-25T22:27:07.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 223338299398 2026-02-25T22:27:07.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-55834574858 2-77309411337 3-270582939651 4-150323855368 5-184683593735 6-223338299398' 2026-02-25T22:27:07.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:07.773 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803786 2026-02-25T22:27:07.773 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:07.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:27:07.775 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803786 2026-02-25T22:27:07.775 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:07.776 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803786 2026-02-25T22:27:07.776 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803786' 2026-02-25T22:27:07.777 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803786 2026-02-25T22:27:07.777 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:27:07.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803786 -lt 25769803786 2026-02-25T22:27:07.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:07.989 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574858 2026-02-25T22:27:07.989 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:07.991 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:27:07.992 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574858 2026-02-25T22:27:07.992 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:07.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574858 2026-02-25T22:27:07.993 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574858 2026-02-25T22:27:07.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574858' 2026-02-25T22:27:07.994 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:27:08.205 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574858 -lt 55834574858 2026-02-25T22:27:08.205 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:08.206 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411337 2026-02-25T22:27:08.206 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:08.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:27:08.208 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411337 2026-02-25T22:27:08.208 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:08.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411337 2026-02-25T22:27:08.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411337' 2026-02-25T22:27:08.209 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 77309411337 2026-02-25T22:27:08.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:27:08.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411337 -lt 77309411337 2026-02-25T22:27:08.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:08.417 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-270582939651 2026-02-25T22:27:08.417 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:08.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:27:08.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-270582939651 2026-02-25T22:27:08.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:08.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=270582939651 2026-02-25T22:27:08.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 270582939651' 2026-02-25T22:27:08.421 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 270582939651 2026-02-25T22:27:08.421 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:27:08.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 270582939651 -lt 270582939651 2026-02-25T22:27:08.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:08.634 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855368 2026-02-25T22:27:08.635 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:08.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:27:08.636 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855368 2026-02-25T22:27:08.637 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:08.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855368 2026-02-25T22:27:08.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855368' 2026-02-25T22:27:08.638 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 150323855368 2026-02-25T22:27:08.639 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:27:08.859 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855368 -lt 150323855368 2026-02-25T22:27:08.859 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:08.860 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-184683593735 2026-02-25T22:27:08.860 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:08.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:27:08.862 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-184683593735 2026-02-25T22:27:08.862 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:08.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593735 2026-02-25T22:27:08.864 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 184683593735 2026-02-25T22:27:08.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 184683593735' 2026-02-25T22:27:08.864 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:27:09.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593735 -lt 184683593735 2026-02-25T22:27:09.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:09.077 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-223338299398 2026-02-25T22:27:09.077 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:09.078 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:27:09.079 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-223338299398 2026-02-25T22:27:09.079 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:09.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=223338299398 2026-02-25T22:27:09.081 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 223338299398 2026-02-25T22:27:09.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 223338299398' 2026-02-25T22:27:09.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:27:09.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 223338299396 -lt 223338299398 2026-02-25T22:27:09.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:27:10.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:27:10.298 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:27:10.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 223338299399 -lt 223338299398 2026-02-25T22:27:10.512 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:27:10.513 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:27:10.513 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:27:10.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:27:10.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:27:10.799 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:27:10.799 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:27:10.799 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:27:10.799 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:27:10.800 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:27:10.800 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:27:11.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:27:11.006 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:27:11.006 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:27:11.006 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:27:11.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:27:11.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:27:11.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:27:11.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:403: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2026-02-25T22:27:11.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:27:11.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:27:11.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2026-02-25T22:27:11.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:27:11.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:27:11.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2026-02-25T22:27:11.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:27:11.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:27:11.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:27:11.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:27:11.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:27:11.559 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:27:11.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:27:11.840 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:27:11.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:27:11.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:27:11.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:27:11.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:27:11.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:27:11.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:27:11.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:27:11.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:27:11.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:27:11.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:27:11.965 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:27:11.966 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:27:11.966 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:27:11.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:27:11.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:27:11.967 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:27:11.968 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:27:11.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:27:11.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:27:11.969 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:27:11.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:27:11.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:27:11.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:27:11.993 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:27:11.994 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:11.994 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:11.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:27:11.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:27:11.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:27:11.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:27:11.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:27:11.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:27:11.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:27:11.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:27:11.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:27:11.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:27:11.997 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:27:11.997 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:27:11.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:27:11.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:27:11.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:27:11.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:27:11.999 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:27:12.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:27:12.000 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:27:12.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:27:12.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:27:12.002 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:27:12.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:27:12.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:27:12.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:27:12.004 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:27:12.004 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:27:12.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:27:12.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:27:12.007 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:27:12.007 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:12.007 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:12.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:27:12.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:27:12.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:27:12.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:27:12.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:27:12.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:12.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:12.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:27:12.011 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:27:12.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:27:12.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:27:12.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:27:12.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:27:12.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:27:12.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:27:12.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:27:12.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:27:12.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:27:12.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:27:12.034 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:27:12.034 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:12.034 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:12.035 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:12.035 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:12.035 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:12.035 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:12.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:27:12.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:27:12.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:27:12.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:12.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:27:12.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:27:12.065 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:27:12.066 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:27:12.066 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:27:12.066 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:27:12.066 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:12.066 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:12.066 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:12.066 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:27:12.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:12.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:27:12.121 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:27:12.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:12.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:27:12.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:27:12.123 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:27:12.123 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:27:12.123 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:27:12.123 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:27:12.123 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:12.123 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:12.123 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:12.123 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:27:12.124 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:12.124 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:27:12.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:27:12.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:27:12.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:27:12.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:27:12.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:27:12.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:27:12.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:27:12.287 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:27:12.288 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:12.288 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:12.288 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:12.289 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:12.289 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:12.289 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:12.289 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:27:12.290 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:27:12.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:27:12.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:27:12.428 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:27:12.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:27:13.438 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:27:13.438 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:27:13.438 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:27:13.438 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:13.439 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:13.439 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:13.439 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:27:13.439 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:27:13.439 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:27:13.488 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:27:13.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:27:13.495 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:27:12.049+0000 7f6ac83caec0 0 load: jerasure load: lrc 2026-02-25T22:27:13.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2026-02-25T22:27:13.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2026-02-25T22:27:13.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:439: TEST_ec_recovery_multiple_errors: local objname=myobject 2026-02-25T22:27:13.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: setup_osds 9 2026-02-25T22:27:13.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2026-02-25T22:27:13.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:27:13.497 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2026-02-25T22:27:13.497 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2026-02-25T22:27:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:27:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:27:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:27:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:27:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:27:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:27:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:27:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:27:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:27:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:27:13.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:13.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:27:13.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:27:13.504 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:27:13.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ac385106-b8d3-4644-8852-20d3cdf65520 2026-02-25T22:27:13.504 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 ac385106-b8d3-4644-8852-20d3cdf65520 2026-02-25T22:27:13.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ac385106-b8d3-4644-8852-20d3cdf65520' 2026-02-25T22:27:13.505 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:27:13.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBd59pRBGQHhAAKPNk07szEwoD8LxsR0Xi8Q== 2026-02-25T22:27:13.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBd59pRBGQHhAAKPNk07szEwoD8LxsR0Xi8Q=="}' 2026-02-25T22:27:13.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ac385106-b8d3-4644-8852-20d3cdf65520 -i td/test-erasure-eio/0/new.json 2026-02-25T22:27:13.622 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:27:13.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:27:13.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBd59pRBGQHhAAKPNk07szEwoD8LxsR0Xi8Q== --osd-uuid ac385106-b8d3-4644-8852-20d3cdf65520 2026-02-25T22:27:13.646 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:13.641+0000 7f2568451a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:13.648 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:13.645+0000 7f2568451a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:13.649 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:13.645+0000 7f2568451a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:13.649 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:13.645+0000 7f2568451a40 -1 bdev(0x560daa3b4400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:27:13.649 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:13.645+0000 7f2568451a40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:27:14.401 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:27:14.401 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:27:14.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:27:14.406 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:27:14.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:27:14.546 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:27:14.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:27:14.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:27:14.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:27:14.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:27:14.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:27:14.563 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:14.561+0000 7f416ca0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:14.564 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:14.561+0000 7f416ca0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:14.565 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:14.561+0000 7f416ca0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:14.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:27:14.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:27:14.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:27:14.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:27:14.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:27:14.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:14.658 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:27:14.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:27:14.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:14.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:27:14.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:14.972 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:14.969+0000 7f416ca0da40 -1 Falling back to public interface 2026-02-25T22:27:15.644 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:15.641+0000 7f416ca0da40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:27:15.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:15.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:15.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:27:15.871 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:27:15.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:15.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:27:16.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:17.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:17.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:17.096 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:27:17.097 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:27:17.097 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:17.097 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:27:17.308 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1892823037,v1:127.0.0.1:6803/1892823037] [v2:127.0.0.1:6804/1892823037,v1:127.0.0.1:6805/1892823037] exists,up ac385106-b8d3-4644-8852-20d3cdf65520 2026-02-25T22:27:17.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:27:17.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:27:17.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:27:17.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:27:17.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:27:17.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:27:17.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:27:17.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:27:17.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:27:17.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:27:17.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:27:17.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:27:17.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:27:17.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:27:17.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:27:17.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:27:17.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:27:17.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:27:17.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:27:17.313 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:27:17.313 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:17.313 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:27:17.314 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:27:17.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ce6539e8-625f-458b-84f8-82cd85cc139e 2026-02-25T22:27:17.315 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 ce6539e8-625f-458b-84f8-82cd85cc139e 2026-02-25T22:27:17.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ce6539e8-625f-458b-84f8-82cd85cc139e' 2026-02-25T22:27:17.315 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:27:17.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBFd59pZvNKExAApX1laDXeFcomz04IQ4yEZg== 2026-02-25T22:27:17.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBFd59pZvNKExAApX1laDXeFcomz04IQ4yEZg=="}' 2026-02-25T22:27:17.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ce6539e8-625f-458b-84f8-82cd85cc139e -i td/test-erasure-eio/1/new.json 2026-02-25T22:27:17.548 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:27:17.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:27:17.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFd59pZvNKExAApX1laDXeFcomz04IQ4yEZg== --osd-uuid ce6539e8-625f-458b-84f8-82cd85cc139e 2026-02-25T22:27:17.576 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:17.573+0000 7f5978984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:17.577 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:17.573+0000 7f5978984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:17.578 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:17.573+0000 7f5978984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:17.579 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:17.577+0000 7f5978984a40 -1 bdev(0x557eb682c400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:27:17.579 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:17.577+0000 7f5978984a40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:27:18.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:27:18.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:27:18.239 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:27:18.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:27:18.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:27:18.526 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:27:18.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:27:18.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:27:18.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:27:18.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:27:18.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:27:18.551 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:18.549+0000 7f0b89d56a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:18.553 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:18.549+0000 7f0b89d56a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:18.554 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:18.549+0000 7f0b89d56a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:18.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:27:18.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:27:18.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:27:18.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:27:18.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:27:18.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:18.731 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:27:18.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:27:18.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:18.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:27:18.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:18.948 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:18.945+0000 7f0b89d56a40 -1 Falling back to public interface 2026-02-25T22:27:19.596 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:19.593+0000 7f0b89d56a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:27:19.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:19.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:19.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:27:19.943 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:27:19.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:19.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:27:20.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:21.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:21.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:21.150 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:27:21.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:27:21.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:21.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:27:21.355 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/719956848,v1:127.0.0.1:6811/719956848] [v2:127.0.0.1:6812/719956848,v1:127.0.0.1:6813/719956848] exists,up ce6539e8-625f-458b-84f8-82cd85cc139e 2026-02-25T22:27:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:27:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:27:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:27:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:27:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:27:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:27:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:27:21.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:27:21.360 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:27:21.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:27:21.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:27:21.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:27:21.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:27:21.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:27:21.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:27:21.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:27:21.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:27:21.363 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:27:21.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c37e22d8-ca19-4309-9a00-6cf0d83f9ddf 2026-02-25T22:27:21.363 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 c37e22d8-ca19-4309-9a00-6cf0d83f9ddf 2026-02-25T22:27:21.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c37e22d8-ca19-4309-9a00-6cf0d83f9ddf' 2026-02-25T22:27:21.364 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:27:21.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBJd59pVZU1FhAAm7+mCDQXEOlOosl0zIGp7w== 2026-02-25T22:27:21.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBJd59pVZU1FhAAm7+mCDQXEOlOosl0zIGp7w=="}' 2026-02-25T22:27:21.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c37e22d8-ca19-4309-9a00-6cf0d83f9ddf -i td/test-erasure-eio/2/new.json 2026-02-25T22:27:21.589 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:27:21.598 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:27:21.599 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJd59pVZU1FhAAm7+mCDQXEOlOosl0zIGp7w== --osd-uuid c37e22d8-ca19-4309-9a00-6cf0d83f9ddf 2026-02-25T22:27:21.616 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:21.613+0000 7f1f97155a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:21.617 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:21.613+0000 7f1f97155a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:21.618 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:21.613+0000 7f1f97155a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:21.619 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:21.617+0000 7f1f97155a40 -1 bdev(0x55e4d0916400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:27:21.619 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:21.617+0000 7f1f97155a40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:27:22.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:27:22.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:27:22.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:27:22.251 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:27:22.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:27:22.529 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:27:22.529 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:27:22.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:27:22.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:27:22.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:27:22.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:27:22.552 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:22.549+0000 7f7d01636a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:22.554 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:22.549+0000 7f7d01636a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:22.555 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:22.553+0000 7f7d01636a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:22.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:27:22.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:27:22.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:27:22.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:27:22.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:27:22.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:22.743 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:27:22.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:27:22.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:22.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:27:22.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:22.972 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:22.969+0000 7f7d01636a40 -1 Falling back to public interface 2026-02-25T22:27:23.669 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:23.665+0000 7f7d01636a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:27:23.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:23.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:23.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:27:23.950 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:27:23.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:23.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:27:24.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:25.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:25.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:25.165 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:27:25.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:27:25.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:25.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:27:25.368 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3353112527,v1:127.0.0.1:6819/3353112527] [v2:127.0.0.1:6820/3353112527,v1:127.0.0.1:6821/3353112527] exists,up c37e22d8-ca19-4309-9a00-6cf0d83f9ddf 2026-02-25T22:27:25.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:27:25.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:27:25.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:27:25.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:27:25.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:27:25.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:27:25.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:27:25.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:27:25.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:27:25.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:27:25.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:27:25.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:25.373 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:27:25.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:27:25.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:27:25.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:27:25.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:27:25.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:27:25.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0d912f28-9bb8-4a4e-a567-bcc88415f4a9 2026-02-25T22:27:25.376 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 0d912f28-9bb8-4a4e-a567-bcc88415f4a9 2026-02-25T22:27:25.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 0d912f28-9bb8-4a4e-a567-bcc88415f4a9' 2026-02-25T22:27:25.376 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:27:25.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBNd59phkXkFhAAC61MUEhALvxvV3mmSHQwrg== 2026-02-25T22:27:25.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBNd59phkXkFhAAC61MUEhALvxvV3mmSHQwrg=="}' 2026-02-25T22:27:25.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0d912f28-9bb8-4a4e-a567-bcc88415f4a9 -i td/test-erasure-eio/3/new.json 2026-02-25T22:27:25.602 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:27:25.611 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:27:25.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNd59phkXkFhAAC61MUEhALvxvV3mmSHQwrg== --osd-uuid 0d912f28-9bb8-4a4e-a567-bcc88415f4a9 2026-02-25T22:27:25.628 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:25.625+0000 7f274ad87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:25.630 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:25.625+0000 7f274ad87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:25.631 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:25.625+0000 7f274ad87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:25.631 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:25.629+0000 7f274ad87a40 -1 bdev(0x55b849ebc400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:27:25.632 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:25.629+0000 7f274ad87a40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:27:26.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:27:26.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:27:26.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:27:26.300 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:27:26.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:27:26.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:27:26.587 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:27:26.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:27:26.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:27:26.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:27:26.589 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:27:26.612 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:26.609+0000 7f03d0349a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:26.613 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:26.609+0000 7f03d0349a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:26.614 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:26.609+0000 7f03d0349a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:26.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:27:26.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:27:26.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:27:26.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:27:26.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:27:26.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:26.795 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:27:26.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:27:26.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:26.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:27:26.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:27.113 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:27.109+0000 7f03d0349a40 -1 Falling back to public interface 2026-02-25T22:27:27.774 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:27.769+0000 7f03d0349a40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:27:27.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:27.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:27.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:27:27.998 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:27:27.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:27.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:27:28.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:29.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:29.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:29.212 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:27:29.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:27:29.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:29.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:27:29.420 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 26 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1498841190,v1:127.0.0.1:6827/1498841190] [v2:127.0.0.1:6828/1498841190,v1:127.0.0.1:6829/1498841190] exists,up 0d912f28-9bb8-4a4e-a567-bcc88415f4a9 2026-02-25T22:27:29.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:27:29.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:27:29.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:27:29.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T22:27:29.424 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:27:29.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:27:29.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:27:29.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:27:29.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:27:29.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:27:29.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:27:29.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:27:29.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:27:29.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T22:27:29.426 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:27:29.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2052d0a2-3c4d-4c46-acf0-960cffd3ec29 2026-02-25T22:27:29.427 INFO:tasks.workunit.client.0.trial125.stdout:add osd4 2052d0a2-3c4d-4c46-acf0-960cffd3ec29 2026-02-25T22:27:29.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 2052d0a2-3c4d-4c46-acf0-960cffd3ec29' 2026-02-25T22:27:29.427 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:27:29.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBRd59pyA3+GRAACazQiSqdowK5u301WFD8vA== 2026-02-25T22:27:29.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBRd59pyA3+GRAACazQiSqdowK5u301WFD8vA=="}' 2026-02-25T22:27:29.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2052d0a2-3c4d-4c46-acf0-960cffd3ec29 -i td/test-erasure-eio/4/new.json 2026-02-25T22:27:29.653 INFO:tasks.workunit.client.0.trial125.stdout:4 2026-02-25T22:27:29.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T22:27:29.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBRd59pyA3+GRAACazQiSqdowK5u301WFD8vA== --osd-uuid 2052d0a2-3c4d-4c46-acf0-960cffd3ec29 2026-02-25T22:27:29.680 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:29.677+0000 7f3ab2de3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:29.681 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:29.677+0000 7f3ab2de3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:29.682 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:29.677+0000 7f3ab2de3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:29.683 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:29.677+0000 7f3ab2de3a40 -1 bdev(0x559f27d76400 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T22:27:29.683 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:29.681+0000 7f3ab2de3a40 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T22:27:30.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T22:27:30.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:27:30.363 INFO:tasks.workunit.client.0.trial125.stdout:adding osd4 key to auth repository 2026-02-25T22:27:30.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T22:27:30.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:27:30.653 INFO:tasks.workunit.client.0.trial125.stdout:start osd.4 2026-02-25T22:27:30.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T22:27:30.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:27:30.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:27:30.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:27:30.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:27:30.669 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:30.665+0000 7faf0d675a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:30.671 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:30.665+0000 7faf0d675a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:30.672 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:30.669+0000 7faf0d675a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:30.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T22:27:30.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:27:30.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T22:27:30.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:27:30.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:27:30.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:30.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:27:30.873 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:27:30.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:30.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:27:31.073 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:31.069+0000 7faf0d675a40 -1 Falling back to public interface 2026-02-25T22:27:31.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:31.776 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:31.773+0000 7faf0d675a40 -1 osd.4 0 log_to_monitors true 2026-02-25T22:27:32.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:32.084 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:32.084 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:27:32.084 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:27:32.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:32.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:27:32.293 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:33.294 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:33.294 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:33.295 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:27:33.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:27:33.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:33.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:27:33.501 INFO:tasks.workunit.client.0.trial125.stdout:osd.4 up in weight 1 up_from 35 up_thru 36 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2391936286,v1:127.0.0.1:6835/2391936286] [v2:127.0.0.1:6836/2391936286,v1:127.0.0.1:6837/2391936286] exists,up 2052d0a2-3c4d-4c46-acf0-960cffd3ec29 2026-02-25T22:27:33.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:27:33.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:27:33.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-25T22:27:33.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:27:33.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:27:33.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:27:33.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:27:33.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:27:33.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:27:33.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:27:33.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:27:33.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:27:33.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:27:33.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:27:33.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-25T22:27:33.508 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:27:33.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c30c734d-bd2a-4b7e-b92e-73cbef8c1645 2026-02-25T22:27:33.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 c30c734d-bd2a-4b7e-b92e-73cbef8c1645' 2026-02-25T22:27:33.508 INFO:tasks.workunit.client.0.trial125.stdout:add osd5 c30c734d-bd2a-4b7e-b92e-73cbef8c1645 2026-02-25T22:27:33.508 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:27:33.518 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBVd59pFpDVHhAApEGmWPRPdt+/9jFrv7dpiA== 2026-02-25T22:27:33.518 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBVd59pFpDVHhAApEGmWPRPdt+/9jFrv7dpiA=="}' 2026-02-25T22:27:33.518 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c30c734d-bd2a-4b7e-b92e-73cbef8c1645 -i td/test-erasure-eio/5/new.json 2026-02-25T22:27:33.734 INFO:tasks.workunit.client.0.trial125.stdout:5 2026-02-25T22:27:33.744 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-25T22:27:33.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVd59pFpDVHhAApEGmWPRPdt+/9jFrv7dpiA== --osd-uuid c30c734d-bd2a-4b7e-b92e-73cbef8c1645 2026-02-25T22:27:33.769 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:33.765+0000 7f788ba94a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:33.771 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:33.765+0000 7f788ba94a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:33.772 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:33.769+0000 7f788ba94a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:33.772 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:33.769+0000 7f788ba94a40 -1 bdev(0x562a1bf30400 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-25T22:27:33.773 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:33.769+0000 7f788ba94a40 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-25T22:27:34.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-25T22:27:34.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:27:34.432 INFO:tasks.workunit.client.0.trial125.stdout:adding osd5 key to auth repository 2026-02-25T22:27:34.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T22:27:34.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:27:34.721 INFO:tasks.workunit.client.0.trial125.stdout:start osd.5 2026-02-25T22:27:34.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T22:27:34.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:27:34.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:27:34.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:27:34.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:27:34.744 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:34.741+0000 7f1b6f8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:34.745 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:34.741+0000 7f1b6f8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:34.746 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:34.741+0000 7f1b6f8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:34.929 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T22:27:34.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:27:34.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T22:27:34.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:27:34.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:27:34.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:34.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:27:34.930 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:27:34.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:34.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:27:35.133 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:35.189 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:35.185+0000 7f1b6f8c4a40 -1 Falling back to public interface 2026-02-25T22:27:35.893 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:35.889+0000 7f1b6f8c4a40 -1 osd.5 0 log_to_monitors true 2026-02-25T22:27:36.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:36.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:36.135 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:27:36.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:27:36.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:36.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:27:36.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:37.349 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:37.349 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:37.349 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:27:37.350 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:27:37.350 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:37.350 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:27:37.559 INFO:tasks.workunit.client.0.trial125.stdout:osd.5 up in weight 1 up_from 44 up_thru 45 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3000488864,v1:127.0.0.1:6843/3000488864] [v2:127.0.0.1:6844/3000488864,v1:127.0.0.1:6845/3000488864] exists,up c30c734d-bd2a-4b7e-b92e-73cbef8c1645 2026-02-25T22:27:37.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:27:37.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:27:37.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:27:37.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:27:37.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-25T22:27:37.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:27:37.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:27:37.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T22:27:37.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:27:37.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:37.564 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:27:37.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:27:37.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:27:37.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:27:37.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:27:37.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-25T22:27:37.566 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:27:37.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=de6e91cd-3c43-419f-b52f-af4779b1aae2 2026-02-25T22:27:37.567 INFO:tasks.workunit.client.0.trial125.stdout:add osd6 de6e91cd-3c43-419f-b52f-af4779b1aae2 2026-02-25T22:27:37.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 de6e91cd-3c43-419f-b52f-af4779b1aae2' 2026-02-25T22:27:37.567 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:27:37.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBZd59pOyFUIhAAxu6r6VULqFyqzmzKze/5dQ== 2026-02-25T22:27:37.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBZd59pOyFUIhAAxu6r6VULqFyqzmzKze/5dQ=="}' 2026-02-25T22:27:37.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new de6e91cd-3c43-419f-b52f-af4779b1aae2 -i td/test-erasure-eio/6/new.json 2026-02-25T22:27:37.784 INFO:tasks.workunit.client.0.trial125.stdout:6 2026-02-25T22:27:37.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-25T22:27:37.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZd59pOyFUIhAAxu6r6VULqFyqzmzKze/5dQ== --osd-uuid de6e91cd-3c43-419f-b52f-af4779b1aae2 2026-02-25T22:27:37.809 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:37.805+0000 7f4247786a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:37.811 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:37.805+0000 7f4247786a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:37.812 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:37.809+0000 7f4247786a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:37.812 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:37.809+0000 7f4247786a40 -1 bdev(0x5650c9928400 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-25T22:27:37.812 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:37.809+0000 7f4247786a40 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-25T22:27:38.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-25T22:27:38.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:27:38.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T22:27:38.436 INFO:tasks.workunit.client.0.trial125.stdout:adding osd6 key to auth repository 2026-02-25T22:27:38.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:27:38.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T22:27:38.732 INFO:tasks.workunit.client.0.trial125.stdout:start osd.6 2026-02-25T22:27:38.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:27:38.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:27:38.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:27:38.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:27:38.748 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:38.745+0000 7fb69e669a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:38.750 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:38.745+0000 7fb69e669a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:38.750 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:38.745+0000 7fb69e669a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:38.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T22:27:38.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:27:38.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T22:27:38.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:27:38.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:27:38.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:38.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:27:38.947 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:27:38.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:38.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:27:39.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:39.201 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:39.197+0000 7fb69e669a40 -1 Falling back to public interface 2026-02-25T22:27:39.841 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:39.836+0000 7fb69e669a40 -1 osd.6 0 log_to_monitors true 2026-02-25T22:27:40.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:40.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:40.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:27:40.155 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:27:40.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:40.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:27:40.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:41.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:41.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:41.371 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:27:41.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:27:41.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:41.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:27:41.575 INFO:tasks.workunit.client.0.trial125.stdout:osd.6 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/4142945127,v1:127.0.0.1:6851/4142945127] [v2:127.0.0.1:6852/4142945127,v1:127.0.0.1:6853/4142945127] exists,up de6e91cd-3c43-419f-b52f-af4779b1aae2 2026-02-25T22:27:41.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/7 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:27:41.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:41.580 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:27:41.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:27:41.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/7 2026-02-25T22:27:41.582 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:27:41.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fa141fe3-700b-4dfc-ae10-9ed27403c25d 2026-02-25T22:27:41.582 INFO:tasks.workunit.client.0.trial125.stdout:add osd7 fa141fe3-700b-4dfc-ae10-9ed27403c25d 2026-02-25T22:27:41.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 fa141fe3-700b-4dfc-ae10-9ed27403c25d' 2026-02-25T22:27:41.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:27:41.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBdd59pmVZFIxAAnFbdIQKVDAXZbNWoAQaUrw== 2026-02-25T22:27:41.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBdd59pmVZFIxAAnFbdIQKVDAXZbNWoAQaUrw=="}' 2026-02-25T22:27:41.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fa141fe3-700b-4dfc-ae10-9ed27403c25d -i td/test-erasure-eio/7/new.json 2026-02-25T22:27:41.809 INFO:tasks.workunit.client.0.trial125.stdout:7 2026-02-25T22:27:41.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/7/new.json 2026-02-25T22:27:41.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdd59pmVZFIxAAnFbdIQKVDAXZbNWoAQaUrw== --osd-uuid fa141fe3-700b-4dfc-ae10-9ed27403c25d 2026-02-25T22:27:41.836 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:41.832+0000 7fa63f6b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:41.837 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:41.832+0000 7fa63f6b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:41.838 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:41.832+0000 7fa63f6b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:41.839 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:41.832+0000 7fa63f6b5a40 -1 bdev(0x564a6f5a2400 td/test-erasure-eio/7/block) open stat got: (1) Operation not permitted 2026-02-25T22:27:41.839 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:41.832+0000 7fa63f6b5a40 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2026-02-25T22:27:42.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2026-02-25T22:27:42.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:27:42.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-25T22:27:42.516 INFO:tasks.workunit.client.0.trial125.stdout:adding osd7 key to auth repository 2026-02-25T22:27:42.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:27:42.802 INFO:tasks.workunit.client.0.trial125.stdout:start osd.7 2026-02-25T22:27:42.802 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-25T22:27:42.802 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:27:42.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:27:42.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:27:42.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:27:42.825 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:42.820+0000 7f60a0f6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:42.827 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:42.824+0000 7f60a0f6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:42.828 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:42.824+0000 7f60a0f6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:43.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-25T22:27:43.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:27:43.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-25T22:27:43.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:27:43.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:27:43.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:43.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:27:43.015 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:27:43.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:43.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-25T22:27:43.222 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:43.277 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:43.272+0000 7f60a0f6da40 -1 Falling back to public interface 2026-02-25T22:27:43.829 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:43.824+0000 7f60a0f6da40 -1 osd.7 0 log_to_monitors true 2026-02-25T22:27:44.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:44.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:44.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:27:44.224 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:27:44.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:44.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-25T22:27:44.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:45.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:45.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:45.428 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:27:45.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:27:45.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:45.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-25T22:27:45.640 INFO:tasks.workunit.client.0.trial125.stdout:osd.7 up in weight 1 up_from 60 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1892387648,v1:127.0.0.1:6859/1892387648] [v2:127.0.0.1:6860/1892387648,v1:127.0.0.1:6861/1892387648] exists,up fa141fe3-700b-4dfc-ae10-9ed27403c25d 2026-02-25T22:27:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:27:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:27:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:27:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:27:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2026-02-25T22:27:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:27:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:27:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-25T22:27:45.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/8 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:27:45.644 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:27:45.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:27:45.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:27:45.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:27:45.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:27:45.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:27:45.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/8 2026-02-25T22:27:45.646 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:27:45.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5abb3dbe-2b9d-407a-999b-1e2ed5fc37a0 2026-02-25T22:27:45.647 INFO:tasks.workunit.client.0.trial125.stdout:add osd8 5abb3dbe-2b9d-407a-999b-1e2ed5fc37a0 2026-02-25T22:27:45.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 5abb3dbe-2b9d-407a-999b-1e2ed5fc37a0' 2026-02-25T22:27:45.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:27:45.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBhd59pyowbJxAA48mAM4RV8cEPlU2GXZ4b7A== 2026-02-25T22:27:45.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBhd59pyowbJxAA48mAM4RV8cEPlU2GXZ4b7A=="}' 2026-02-25T22:27:45.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5abb3dbe-2b9d-407a-999b-1e2ed5fc37a0 -i td/test-erasure-eio/8/new.json 2026-02-25T22:27:45.876 INFO:tasks.workunit.client.0.trial125.stdout:8 2026-02-25T22:27:45.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/8/new.json 2026-02-25T22:27:45.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhd59pyowbJxAA48mAM4RV8cEPlU2GXZ4b7A== --osd-uuid 5abb3dbe-2b9d-407a-999b-1e2ed5fc37a0 2026-02-25T22:27:45.903 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:45.900+0000 7f7db55ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:45.905 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:45.900+0000 7f7db55ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:45.906 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:45.900+0000 7f7db55ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:45.906 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:45.900+0000 7f7db55ffa40 -1 bdev(0x562366ff8400 td/test-erasure-eio/8/block) open stat got: (1) Operation not permitted 2026-02-25T22:27:45.906 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:45.900+0000 7f7db55ffa40 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2026-02-25T22:27:46.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2026-02-25T22:27:46.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:27:46.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-25T22:27:46.585 INFO:tasks.workunit.client.0.trial125.stdout:adding osd8 key to auth repository 2026-02-25T22:27:46.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:27:46.867 INFO:tasks.workunit.client.0.trial125.stdout:start osd.8 2026-02-25T22:27:46.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-25T22:27:46.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:27:46.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:27:46.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:27:46.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:27:46.890 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:46.884+0000 7f8b70bdda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:46.892 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:46.888+0000 7f8b70bdda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:46.893 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:46.888+0000 7f8b70bdda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:27:47.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-25T22:27:47.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:27:47.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-25T22:27:47.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:27:47.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:27:47.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:47.075 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:27:47.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:27:47.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:47.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-25T22:27:47.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:47.393 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:47.388+0000 7f8b70bdda40 -1 Falling back to public interface 2026-02-25T22:27:48.029 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:27:48.024+0000 7f8b70bdda40 -1 osd.8 0 log_to_monitors true 2026-02-25T22:27:48.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:48.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:48.284 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:27:48.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:27:48.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:48.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-25T22:27:48.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:27:49.487 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:27:49.487 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:27:49.487 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:27:49.487 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:27:49.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:27:49.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-25T22:27:49.697 INFO:tasks.workunit.client.0.trial125.stdout:osd.8 up in weight 1 up_from 67 up_thru 67 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3146497941,v1:127.0.0.1:6867/3146497941] [v2:127.0.0.1:6868/3146497941,v1:127.0.0.1:6869/3146497941] exists,up 5abb3dbe-2b9d-407a-999b-1e2ed5fc37a0 2026-02-25T22:27:49.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:27:49.698 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:27:49.698 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:27:49.698 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:27:49.698 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:27:49.698 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:27:49.699 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:49.699 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:49.699 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:49.699 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:27:49.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:27:49.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:27:49.753 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:27:49.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:27:49.762 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:27:14.973+0000 7f416ca0da40 0 load: jerasure load: lrc 2026-02-25T22:27:49.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2026-02-25T22:27:49.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:444: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2026-02-25T22:27:49.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:27:49.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:27:49.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2026-02-25T22:27:49.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:27:49.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2026-02-25T22:27:49.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:27:49.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2026-02-25T22:27:50.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:27:50.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:27:50.383 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:27:50.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:27:51.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:27:51.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:27:51.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:27:51.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:27:51.394 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:27:51.395 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:27:51.395 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:27:51.395 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:27:51.395 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:27:51.395 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:27:51.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:27:51.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:27:51.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:27:51.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:27:51.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:27:51.501 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:27:51.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:27:51.714 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:27:51.714 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:27:51.715 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:27:51.715 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:27:51.715 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:27:51.715 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:27:51.715 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:27:51.715 INFO:tasks.workunit.client.0.trial125.stderr:8' 2026-02-25T22:27:51.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:27:51.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:51.715 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:27:51.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803785 2026-02-25T22:27:51.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803785 2026-02-25T22:27:51.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785' 2026-02-25T22:27:51.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:51.796 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:27:51.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574857 2026-02-25T22:27:51.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574857 2026-02-25T22:27:51.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857' 2026-02-25T22:27:51.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:51.873 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:27:51.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378632 2026-02-25T22:27:51.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378632 2026-02-25T22:27:51.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632' 2026-02-25T22:27:51.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:51.952 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:27:52.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149703 2026-02-25T22:27:52.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149703 2026-02-25T22:27:52.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632 3-111669149703' 2026-02-25T22:27:52.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:52.031 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:27:52.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855366 2026-02-25T22:27:52.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855366 2026-02-25T22:27:52.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632 3-111669149703 4-150323855366' 2026-02-25T22:27:52.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:52.110 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:27:52.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561030 2026-02-25T22:27:52.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561030 2026-02-25T22:27:52.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632 3-111669149703 4-150323855366 5-188978561030' 2026-02-25T22:27:52.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:52.191 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:27:52.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=223338299397 2026-02-25T22:27:52.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 223338299397 2026-02-25T22:27:52.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632 3-111669149703 4-150323855366 5-188978561030 6-223338299397' 2026-02-25T22:27:52.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:52.271 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T22:27:52.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037764 2026-02-25T22:27:52.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037764 2026-02-25T22:27:52.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632 3-111669149703 4-150323855366 5-188978561030 6-223338299397 7-257698037764' 2026-02-25T22:27:52.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:52.348 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T22:27:52.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808836 2026-02-25T22:27:52.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808836 2026-02-25T22:27:52.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632 3-111669149703 4-150323855366 5-188978561030 6-223338299397 7-257698037764 8-287762808836' 2026-02-25T22:27:52.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:52.427 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803785 2026-02-25T22:27:52.427 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:52.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:27:52.428 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803785 2026-02-25T22:27:52.429 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:52.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803785 2026-02-25T22:27:52.430 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803785 2026-02-25T22:27:52.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2026-02-25T22:27:52.431 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:27:52.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803786 -lt 25769803785 2026-02-25T22:27:52.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:52.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574857 2026-02-25T22:27:52.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:52.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:27:52.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574857 2026-02-25T22:27:52.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:52.649 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574857 2026-02-25T22:27:52.649 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574857 2026-02-25T22:27:52.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574857' 2026-02-25T22:27:52.650 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:27:52.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574857 -lt 55834574857 2026-02-25T22:27:52.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:52.874 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378632 2026-02-25T22:27:52.874 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:52.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:27:52.875 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378632 2026-02-25T22:27:52.876 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:52.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378632 2026-02-25T22:27:52.878 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 81604378632 2026-02-25T22:27:52.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378632' 2026-02-25T22:27:52.878 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:27:53.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378632 -lt 81604378632 2026-02-25T22:27:53.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:53.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149703 2026-02-25T22:27:53.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:53.097 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:27:53.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149703 2026-02-25T22:27:53.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:53.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149703 2026-02-25T22:27:53.099 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149703 2026-02-25T22:27:53.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149703' 2026-02-25T22:27:53.100 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:27:53.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149703 -lt 111669149703 2026-02-25T22:27:53.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:53.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855366 2026-02-25T22:27:53.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:53.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:27:53.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855366 2026-02-25T22:27:53.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:53.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855366 2026-02-25T22:27:53.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855366' 2026-02-25T22:27:53.312 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 150323855366 2026-02-25T22:27:53.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:27:53.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855366 -lt 150323855366 2026-02-25T22:27:53.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:53.530 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-188978561030 2026-02-25T22:27:53.531 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:53.531 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:27:53.532 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-188978561030 2026-02-25T22:27:53.533 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:53.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561030 2026-02-25T22:27:53.534 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 188978561030 2026-02-25T22:27:53.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 188978561030' 2026-02-25T22:27:53.534 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:27:53.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561030 -lt 188978561030 2026-02-25T22:27:53.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:53.741 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-223338299397 2026-02-25T22:27:53.742 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:53.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:27:53.744 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-223338299397 2026-02-25T22:27:53.744 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:53.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=223338299397 2026-02-25T22:27:53.745 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 223338299397 2026-02-25T22:27:53.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 223338299397' 2026-02-25T22:27:53.746 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:27:53.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 223338299397 -lt 223338299397 2026-02-25T22:27:53.977 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:53.977 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-257698037764 2026-02-25T22:27:53.977 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:53.979 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T22:27:53.979 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-257698037764 2026-02-25T22:27:53.979 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:53.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=257698037764 2026-02-25T22:27:53.981 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.7 seq 257698037764 2026-02-25T22:27:53.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 257698037764' 2026-02-25T22:27:53.982 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T22:27:54.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 257698037764 -lt 257698037764 2026-02-25T22:27:54.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:54.199 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-287762808836 2026-02-25T22:27:54.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:54.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T22:27:54.202 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-287762808836 2026-02-25T22:27:54.202 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:54.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=287762808836 2026-02-25T22:27:54.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 287762808836' 2026-02-25T22:27:54.204 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.8 seq 287762808836 2026-02-25T22:27:54.204 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T22:27:54.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 287762808836 -lt 287762808836 2026-02-25T22:27:54.418 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:27:54.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:27:54.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:27:54.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:27:54.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:27:54.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:27:54.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:27:54.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:27:54.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:27:54.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:27:54.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:27:54.907 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:27:54.908 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:27:54.908 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:27:54.908 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:27:55.193 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:27:55.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:27:55.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:27:55.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:446: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-25T22:27:55.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:27:55.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:27:55.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-25T22:27:55.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:27:55.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:27:55.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:27:55.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:27:55.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:27:55.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:27:55.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:27:55.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:27:55.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-25T22:27:55.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:447: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2026-02-25T22:27:55.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:27:55.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:27:55.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:27:55.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:27:55.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:27:55.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:27:55.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T22:27:55.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:27:55.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:27:55.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:27:55.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T22:27:55.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:27:55.215 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T22:27:55.215 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:27:55.215 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T22:27:55.216 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:27:55.216 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:27:55.428 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:27:55.428 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:27:55.428 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:27:55.428 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:27:55.428 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:27:55.428 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:27:55.428 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:27:55.428 INFO:tasks.workunit.client.0.trial125.stderr:7' 2026-02-25T22:27:55.429 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-25T22:27:55.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-25T22:27:55.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:27:55.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T22:27:55.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:27:55.430 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T22:27:55.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:27:55.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T22:27:55.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:27:55.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T22:27:55.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:27:55.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:27:55.431 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:27:55.431 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T22:27:55.431 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:27:55.431 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:27:55.432 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:55.432 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:55.432 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:55.432 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:27:55.433 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:27:55.492 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:27:55.492 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:27:55.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:27:55.493 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T22:27:55.493 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:27:55.493 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:27:55.494 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:55.494 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:55.494 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:55.495 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:27:55.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:27:55.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2026-02-25T22:27:55.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:450: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2026-02-25T22:27:55.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:27:55.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:27:55.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:27:55.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:27:55.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:27:55.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:27:55.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T22:27:55.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:27:55.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:27:55.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:27:55.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=3 2026-02-25T22:27:55.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:27:55.558 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T22:27:55.558 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:27:55.558 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T22:27:55.559 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:27:55.559 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:27:55.770 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:27:55.770 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:27:55.770 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:27:55.771 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:27:55.771 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:27:55.771 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:27:55.771 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:27:55.771 INFO:tasks.workunit.client.0.trial125.stderr:7' 2026-02-25T22:27:55.771 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-25T22:27:55.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-25T22:27:55.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:27:55.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=0 2026-02-25T22:27:55.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:27:55.773 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/0/type 2026-02-25T22:27:55.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:27:55.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-25T22:27:55.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:27:55.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-25T22:27:55.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:27:55.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:27:55.773 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:27:55.773 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-25T22:27:55.773 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:27:55.773 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:27:55.774 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:55.774 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:55.774 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:55.775 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:27:55.775 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:27:55.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:27:55.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:27:55.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:27:55.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.0 2026-02-25T22:27:55.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:27:55.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:27:55.836 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:55.836 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:55.837 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:55.837 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:27:55.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:27:55.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2026-02-25T22:27:55.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2026-02-25T22:27:55.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:27:55.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:27:55.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:27:55.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:27:55.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:27:55.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:27:55.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T22:27:55.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:27:55.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:27:55.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:27:55.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=4 2026-02-25T22:27:55.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:27:55.896 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T22:27:55.896 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:27:55.896 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T22:27:55.897 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:27:55.897 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:27:56.114 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:27:56.114 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:27:56.114 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:27:56.114 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:27:56.115 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:27:56.115 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:27:56.115 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:27:56.115 INFO:tasks.workunit.client.0.trial125.stderr:7' 2026-02-25T22:27:56.115 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-25T22:27:56.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-25T22:27:56.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:27:56.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=6 2026-02-25T22:27:56.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:27:56.115 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/6/type 2026-02-25T22:27:56.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:27:56.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2026-02-25T22:27:56.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:27:56.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=6 2026-02-25T22:27:56.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:27:56.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:27:56.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:27:56.117 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.6 2026-02-25T22:27:56.117 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.6 2026-02-25T22:27:56.117 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.6 ']' 2026-02-25T22:27:56.118 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:56.118 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:56.118 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:56.118 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.6.asok 2026-02-25T22:27:56.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:27:56.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:27:56.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:27:56.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:27:56.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.6 2026-02-25T22:27:56.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.6 2026-02-25T22:27:56.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.6 ']' 2026-02-25T22:27:56.180 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:56.180 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:56.180 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:27:56.180 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.6.asok 2026-02-25T22:27:56.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:27:56.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2026-02-25T22:27:56.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2026-02-25T22:27:56.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:27:56.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T22:27:56.239 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:27:56.239 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:27:56.453 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:27:56.453 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:27:56.453 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:27:56.454 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:27:56.454 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:27:56.454 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:27:56.454 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:27:56.454 INFO:tasks.workunit.client.0.trial125.stderr:7' 2026-02-25T22:27:56.454 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-25T22:27:56.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-25T22:27:56.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: local -a initial_osds 2026-02-25T22:27:56.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_errors: local last_osd=7 2026-02-25T22:27:56.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2026-02-25T22:27:56.456 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:27:56.456 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:27:56.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:27:56.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:27:56.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:27:56.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:27:56.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_errors: ceph osd down 7 2026-02-25T22:27:56.965 INFO:tasks.workunit.client.0.trial125.stderr:osd.7 is already down. 2026-02-25T22:27:56.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:458: TEST_ec_recovery_multiple_errors: ceph osd out 7 2026-02-25T22:27:57.220 INFO:tasks.workunit.client.0.trial125.stderr:osd.7 is already out. 2026-02-25T22:27:57.230 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:461: TEST_ec_recovery_multiple_errors: wait_for_clean 2026-02-25T22:27:57.230 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:27:57.230 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:27:57.230 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:27:57.230 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:27:57.231 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:27:57.231 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:27:57.231 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:27:57.231 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:27:57.232 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:27:57.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:27:57.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:27:57.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:27:57.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:27:57.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:27:57.340 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:27:57.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:27:57.555 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:27:57.555 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:27:57.555 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:27:57.555 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:27:57.556 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:27:57.556 INFO:tasks.workunit.client.0.trial125.stderr:6 2026-02-25T22:27:57.556 INFO:tasks.workunit.client.0.trial125.stderr:7 2026-02-25T22:27:57.556 INFO:tasks.workunit.client.0.trial125.stderr:8' 2026-02-25T22:27:57.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:27:57.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:57.556 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:27:57.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803789 2026-02-25T22:27:57.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803789 2026-02-25T22:27:57.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789' 2026-02-25T22:27:57.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:57.640 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:27:57.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574860 2026-02-25T22:27:57.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574860 2026-02-25T22:27:57.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789 1-55834574860' 2026-02-25T22:27:57.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:57.717 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:27:57.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378635 2026-02-25T22:27:57.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378635 2026-02-25T22:27:57.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789 1-55834574860 2-81604378635' 2026-02-25T22:27:57.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:57.800 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:27:57.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149706 2026-02-25T22:27:57.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149706 2026-02-25T22:27:57.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789 1-55834574860 2-81604378635 3-111669149706' 2026-02-25T22:27:57.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:57.878 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:27:57.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855370 2026-02-25T22:27:57.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855370 2026-02-25T22:27:57.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789 1-55834574860 2-81604378635 3-111669149706 4-150323855370' 2026-02-25T22:27:57.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:57.957 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:27:58.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561033 2026-02-25T22:27:58.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561033 2026-02-25T22:27:58.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789 1-55834574860 2-81604378635 3-111669149706 4-150323855370 5-188978561033' 2026-02-25T22:27:58.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:58.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:27:58.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=223338299400 2026-02-25T22:27:58.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 223338299400 2026-02-25T22:27:58.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789 1-55834574860 2-81604378635 3-111669149706 4-150323855370 5-188978561033 6-223338299400' 2026-02-25T22:27:58.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:58.118 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T22:27:58.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq= 2026-02-25T22:27:58.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: true 2026-02-25T22:27:58.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z '' 2026-02-25T22:27:58.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: continue 2026-02-25T22:27:58.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:27:58.178 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T22:27:58.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808839 2026-02-25T22:27:58.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808839 2026-02-25T22:27:58.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789 1-55834574860 2-81604378635 3-111669149706 4-150323855370 5-188978561033 6-223338299400 8-287762808839' 2026-02-25T22:27:58.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:58.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803789 2026-02-25T22:27:58.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:58.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:27:58.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803789 2026-02-25T22:27:58.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:58.255 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803789 2026-02-25T22:27:58.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803789' 2026-02-25T22:27:58.256 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803789 2026-02-25T22:27:58.256 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:27:58.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803786 -lt 25769803789 2026-02-25T22:27:58.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:27:59.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:27:59.471 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:27:59.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803789 -lt 25769803789 2026-02-25T22:27:59.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:59.679 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574860 2026-02-25T22:27:59.679 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:59.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:27:59.681 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574860 2026-02-25T22:27:59.682 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:59.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574860 2026-02-25T22:27:59.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574860' 2026-02-25T22:27:59.683 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574860 2026-02-25T22:27:59.684 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:27:59.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574860 -lt 55834574860 2026-02-25T22:27:59.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:27:59.900 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378635 2026-02-25T22:27:59.901 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:27:59.902 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:27:59.902 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378635 2026-02-25T22:27:59.903 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:27:59.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378635 2026-02-25T22:27:59.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378635' 2026-02-25T22:27:59.904 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 81604378635 2026-02-25T22:27:59.904 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:28:00.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378635 -lt 81604378635 2026-02-25T22:28:00.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:00.113 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149706 2026-02-25T22:28:00.113 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:00.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:28:00.115 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149706 2026-02-25T22:28:00.115 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:00.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149706 2026-02-25T22:28:00.117 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149706 2026-02-25T22:28:00.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149706' 2026-02-25T22:28:00.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:28:00.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149706 -lt 111669149706 2026-02-25T22:28:00.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:00.342 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855370 2026-02-25T22:28:00.342 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:00.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:28:00.344 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855370 2026-02-25T22:28:00.344 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:00.346 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855370 2026-02-25T22:28:00.346 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 150323855370 2026-02-25T22:28:00.346 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855370' 2026-02-25T22:28:00.346 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:28:00.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855370 -lt 150323855370 2026-02-25T22:28:00.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:00.567 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-188978561033 2026-02-25T22:28:00.567 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:00.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:28:00.569 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-188978561033 2026-02-25T22:28:00.569 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:00.570 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561033 2026-02-25T22:28:00.570 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 188978561033 2026-02-25T22:28:00.571 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 188978561033' 2026-02-25T22:28:00.571 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:28:00.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561033 -lt 188978561033 2026-02-25T22:28:00.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:00.790 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-223338299400 2026-02-25T22:28:00.790 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:00.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:28:00.792 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-223338299400 2026-02-25T22:28:00.792 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:00.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=223338299400 2026-02-25T22:28:00.794 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 223338299400 2026-02-25T22:28:00.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 223338299400' 2026-02-25T22:28:00.794 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:28:01.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 223338299400 -lt 223338299400 2026-02-25T22:28:01.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:01.017 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-287762808839 2026-02-25T22:28:01.017 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:01.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T22:28:01.019 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-287762808839 2026-02-25T22:28:01.019 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:01.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=287762808839 2026-02-25T22:28:01.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 287762808839' 2026-02-25T22:28:01.021 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.8 seq 287762808839 2026-02-25T22:28:01.021 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T22:28:01.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 287762808840 -lt 287762808839 2026-02-25T22:28:01.243 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:28:01.244 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:28:01.244 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:28:01.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:28:01.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:28:01.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:28:01.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:28:01.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:28:01.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:28:01.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:28:01.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:28:01.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:28:01.738 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:28:01.738 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:28:01.738 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:28:02.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:28:02.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:28:02.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:28:02.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2026-02-25T22:28:02.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:28:02.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:28:02.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2026-02-25T22:28:02.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:28:02.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:28:02.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2026-02-25T22:28:02.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:28:02.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:28:02.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:465: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:28:02.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:28:02.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:28:02.312 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:28:02.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:28:02.596 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:28:02.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:28:02.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:28:02.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:02.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:28:02.608 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:02.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:02.609 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:02.609 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:02.609 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:02.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:02.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:02.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:02.723 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:02.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:02.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:02.725 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:02.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:02.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:02.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:02.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:02.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:02.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:02.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:28:02.755 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:02.755 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:02.755 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:02.755 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:28:02.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:02.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:02.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:28:02.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:28:02.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:28:02.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:28:02.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:28:02.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:02.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:28:02.758 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:02.759 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:02.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:02.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:02.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:02.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:02.760 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:02.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:02.761 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:02.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:02.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:02.763 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:02.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:02.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:02.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:02.765 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:02.766 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:02.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:02.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:28:02.768 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:02.768 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:02.768 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:02.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:28:02.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:02.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:02.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:28:02.771 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:02.771 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:02.771 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:02.771 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:28:02.773 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:02.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:28:02.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:02.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:28:02.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:28:02.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:28:02.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:28:02.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:28:02.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:28:02.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:28:02.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:28:02.795 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:28:02.795 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:02.795 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:02.796 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:02.796 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:02.796 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:02.796 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:02.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:28:02.825 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:28:02.826 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:28:02.826 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:02.826 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:02.826 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:28:02.827 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:28:02.827 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:02.827 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:02.827 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:02.827 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:02.827 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:02.827 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:02.828 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:28:02.828 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:02.828 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:28:02.882 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:28:02.882 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:02.882 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:02.882 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:28:02.883 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:28:02.883 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:02.883 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:02.883 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:02.883 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:02.883 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:02.884 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:02.884 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:28:02.884 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:02.884 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:28:02.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:28:02.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:28:02.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:28:02.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:28:02.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:28:02.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:28:02.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:28:03.047 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:28:03.047 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:03.047 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:03.048 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:03.048 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:03.048 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:03.048 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:03.049 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:03.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:03.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:28:03.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:28:03.174 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:28:03.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:28:04.184 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:28:04.184 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:04.184 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:04.185 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:04.185 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:04.185 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:04.185 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:28:04.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:28:04.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:28:04.234 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:28:04.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:28:04.241 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:28:02.808+0000 7f41798a7ec0 0 load: jerasure load: lrc 2026-02-25T22:28:04.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2026-02-25T22:28:04.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2026-02-25T22:28:04.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: local objname=myobject 2026-02-25T22:28:04.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:474: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:28:04.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:04.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:476: TEST_ec_recovery_multiple_objects: setup_osds 7 2026-02-25T22:28:04.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-25T22:28:04.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:28:04.244 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-25T22:28:04.247 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-25T22:28:04.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:28:04.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:28:04.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:28:04.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:04.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:28:04.248 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:28:04.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:04.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:04.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:04.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:04.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:04.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:04.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:28:04.251 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:04.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fb2ac0ee-447c-4be0-b62d-8dc63285fc7b 2026-02-25T22:28:04.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 fb2ac0ee-447c-4be0-b62d-8dc63285fc7b' 2026-02-25T22:28:04.251 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 fb2ac0ee-447c-4be0-b62d-8dc63285fc7b 2026-02-25T22:28:04.251 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:04.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB0d59pOaWFDxAAHLaNibVG8pUAfq1+E9/tzw== 2026-02-25T22:28:04.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB0d59pOaWFDxAAHLaNibVG8pUAfq1+E9/tzw=="}' 2026-02-25T22:28:04.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fb2ac0ee-447c-4be0-b62d-8dc63285fc7b -i td/test-erasure-eio/0/new.json 2026-02-25T22:28:04.372 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:04.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:28:04.381 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0d59pOaWFDxAAHLaNibVG8pUAfq1+E9/tzw== --osd-uuid fb2ac0ee-447c-4be0-b62d-8dc63285fc7b 2026-02-25T22:28:04.398 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:04.392+0000 7f1339b54a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:04.400 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:04.396+0000 7f1339b54a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:04.401 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:04.396+0000 7f1339b54a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:04.401 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:04.396+0000 7f1339b54a40 -1 bdev(0x555c6149a400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:04.401 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:04.396+0000 7f1339b54a40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:28:05.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:28:05.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:05.163 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:28:05.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:28:05.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:05.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:28:05.302 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:28:05.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:05.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:05.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:05.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:05.320 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:05.316+0000 7f02d7ed0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:05.322 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:05.316+0000 7f02d7ed0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:05.322 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:05.316+0000 7f02d7ed0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:05.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:28:05.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:05.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:05.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:05.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:05.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:05.419 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:05.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:05.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:05.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:05.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:05.749 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:05.744+0000 7f02d7ed0a40 -1 Falling back to public interface 2026-02-25T22:28:06.389 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:06.384+0000 7f02d7ed0a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:28:06.632 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:06.632 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:06.632 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:06.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:06.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:06.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:06.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:07.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:07.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:07.851 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:28:07.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:07.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:07.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:08.068 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1564622803,v1:127.0.0.1:6803/1564622803] [v2:127.0.0.1:6804/1564622803,v1:127.0.0.1:6805/1564622803] exists,up fb2ac0ee-447c-4be0-b62d-8dc63285fc7b 2026-02-25T22:28:08.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:08.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:08.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:08.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:28:08.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:28:08.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:28:08.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:08.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:28:08.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:08.073 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:08.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:08.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:08.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:08.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:08.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:28:08.076 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:08.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fad1c3ed-6e40-4ecb-bb1f-346bbeeefd3f 2026-02-25T22:28:08.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 fad1c3ed-6e40-4ecb-bb1f-346bbeeefd3f' 2026-02-25T22:28:08.076 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 fad1c3ed-6e40-4ecb-bb1f-346bbeeefd3f 2026-02-25T22:28:08.076 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:08.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB4d59pqgYZBRAABVETJJTehLBiWDED9eD3vw== 2026-02-25T22:28:08.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB4d59pqgYZBRAABVETJJTehLBiWDED9eD3vw=="}' 2026-02-25T22:28:08.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fad1c3ed-6e40-4ecb-bb1f-346bbeeefd3f -i td/test-erasure-eio/1/new.json 2026-02-25T22:28:08.309 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:08.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:28:08.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB4d59pqgYZBRAABVETJJTehLBiWDED9eD3vw== --osd-uuid fad1c3ed-6e40-4ecb-bb1f-346bbeeefd3f 2026-02-25T22:28:08.338 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:08.332+0000 7fc460cb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:08.340 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:08.336+0000 7fc460cb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:08.341 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:08.336+0000 7fc460cb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:08.341 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:08.336+0000 7fc460cb8a40 -1 bdev(0x5588debc2400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:08.341 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:08.336+0000 7fc460cb8a40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:28:08.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:28:08.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:08.979 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:28:08.979 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:28:08.979 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:09.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:28:09.261 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:28:09.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:09.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:09.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:09.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:09.286 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:09.280+0000 7f67c5f83a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:09.288 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:09.284+0000 7f67c5f83a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:09.288 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:09.284+0000 7f67c5f83a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:09.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:28:09.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:09.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:28:09.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:09.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:09.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:09.470 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:09.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:09.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:09.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:28:09.661 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:09.656+0000 7f67c5f83a40 -1 Falling back to public interface 2026-02-25T22:28:09.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:10.332 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:10.328+0000 7f67c5f83a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:28:10.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:10.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:10.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:10.681 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:10.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:28:10.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:11.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:11.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:11.905 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:28:11.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:11.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:11.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:28:12.116 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2395371778,v1:127.0.0.1:6811/2395371778] [v2:127.0.0.1:6812/2395371778,v1:127.0.0.1:6813/2395371778] exists,up fad1c3ed-6e40-4ecb-bb1f-346bbeeefd3f 2026-02-25T22:28:12.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:12.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:12.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:12.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:12.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:12.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:12.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:12.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:28:12.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:28:12.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:12.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:12.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:12.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:12.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:12.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:12.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:28:12.123 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:12.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=81d1a608-2fd8-45c3-8597-9c632dcb14d9 2026-02-25T22:28:12.123 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 81d1a608-2fd8-45c3-8597-9c632dcb14d9 2026-02-25T22:28:12.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 81d1a608-2fd8-45c3-8597-9c632dcb14d9' 2026-02-25T22:28:12.124 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:12.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB8d59peuX0BxAAwzXQmDZTTsxEEwl9lx1Urw== 2026-02-25T22:28:12.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB8d59peuX0BxAAwzXQmDZTTsxEEwl9lx1Urw=="}' 2026-02-25T22:28:12.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 81d1a608-2fd8-45c3-8597-9c632dcb14d9 -i td/test-erasure-eio/2/new.json 2026-02-25T22:28:12.359 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:28:12.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:28:12.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8d59peuX0BxAAwzXQmDZTTsxEEwl9lx1Urw== --osd-uuid 81d1a608-2fd8-45c3-8597-9c632dcb14d9 2026-02-25T22:28:12.388 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:12.384+0000 7f7543526a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:12.390 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:12.384+0000 7f7543526a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:12.391 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:12.384+0000 7f7543526a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:12.391 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:12.388+0000 7f7543526a40 -1 bdev(0x55bbaf298400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:12.391 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:12.388+0000 7f7543526a40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:28:12.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:28:12.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:12.995 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:28:12.995 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:28:12.995 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:13.294 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:28:13.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:28:13.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:13.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:13.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:13.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:13.322 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:13.316+0000 7fac4c40ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:13.324 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:13.320+0000 7fac4c40ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:13.325 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:13.320+0000 7fac4c40ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:13.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:28:13.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:13.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:28:13.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:13.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:13.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:13.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:13.506 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:13.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:13.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:28:13.721 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:13.716+0000 7fac4c40ba40 -1 Falling back to public interface 2026-02-25T22:28:13.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:14.329 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:14.324+0000 7fac4c40ba40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:28:14.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:14.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:14.723 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:14.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:14.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:14.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:28:14.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:15.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:15.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:15.935 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:28:15.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:15.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:15.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:28:16.142 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3658784549,v1:127.0.0.1:6819/3658784549] [v2:127.0.0.1:6820/3658784549,v1:127.0.0.1:6821/3658784549] exists,up 81d1a608-2fd8-45c3-8597-9c632dcb14d9 2026-02-25T22:28:16.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:16.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:16.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:16.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:28:16.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:28:16.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:28:16.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:16.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:28:16.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:28:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:28:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:28:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:16.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:28:16.147 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:28:16.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:16.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:16.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:16.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:16.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:16.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:16.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:28:16.149 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:16.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f9ba3935-9532-427b-a14f-00bb001094ad 2026-02-25T22:28:16.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 f9ba3935-9532-427b-a14f-00bb001094ad' 2026-02-25T22:28:16.150 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 f9ba3935-9532-427b-a14f-00bb001094ad 2026-02-25T22:28:16.150 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:16.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCAd59py/N9CRAA/MXldtGJApfL1Ir1OAWtWQ== 2026-02-25T22:28:16.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCAd59py/N9CRAA/MXldtGJApfL1Ir1OAWtWQ=="}' 2026-02-25T22:28:16.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f9ba3935-9532-427b-a14f-00bb001094ad -i td/test-erasure-eio/3/new.json 2026-02-25T22:28:16.379 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:28:16.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:28:16.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCAd59py/N9CRAA/MXldtGJApfL1Ir1OAWtWQ== --osd-uuid f9ba3935-9532-427b-a14f-00bb001094ad 2026-02-25T22:28:16.414 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:16.408+0000 7fb30c2f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:16.416 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:16.412+0000 7fb30c2f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:16.417 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:16.412+0000 7fb30c2f3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:16.418 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:16.412+0000 7fb30c2f3a40 -1 bdev(0x56471a81c400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:16.418 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:16.412+0000 7fb30c2f3a40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:28:17.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:28:17.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:17.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:28:17.052 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:28:17.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:17.337 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:28:17.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:28:17.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:17.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:17.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:17.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:17.363 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:17.360+0000 7fb6b449ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:17.365 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:17.360+0000 7fb6b449ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:17.366 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:17.360+0000 7fb6b449ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:17.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:28:17.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:17.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:28:17.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:17.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:17.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:17.551 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:17.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:17.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:17.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:28:17.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:17.769 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:17.764+0000 7fb6b449ca40 -1 Falling back to public interface 2026-02-25T22:28:18.360 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:18.356+0000 7fb6b449ca40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:28:18.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:18.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:18.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:18.763 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:18.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:18.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:28:18.996 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3663990973,v1:127.0.0.1:6827/3663990973] [v2:127.0.0.1:6828/3663990973,v1:127.0.0.1:6829/3663990973] exists,up f9ba3935-9532-427b-a14f-00bb001094ad 2026-02-25T22:28:18.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:18.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:18.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:18.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:28:18.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T22:28:18.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:28:18.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:18.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:28:19.000 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:28:19.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:28:19.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:19.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:19.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:19.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:19.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:19.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:19.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T22:28:19.003 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:19.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=91b8d0ad-91d5-4e66-8994-4bb462627f30 2026-02-25T22:28:19.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 91b8d0ad-91d5-4e66-8994-4bb462627f30' 2026-02-25T22:28:19.003 INFO:tasks.workunit.client.0.trial125.stdout:add osd4 91b8d0ad-91d5-4e66-8994-4bb462627f30 2026-02-25T22:28:19.003 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:19.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCDd59pb7TFABAAeJGnLfuE6O1c/Z2YQ0Y/jw== 2026-02-25T22:28:19.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCDd59pb7TFABAAeJGnLfuE6O1c/Z2YQ0Y/jw=="}' 2026-02-25T22:28:19.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 91b8d0ad-91d5-4e66-8994-4bb462627f30 -i td/test-erasure-eio/4/new.json 2026-02-25T22:28:19.240 INFO:tasks.workunit.client.0.trial125.stdout:4 2026-02-25T22:28:19.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T22:28:19.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCDd59pb7TFABAAeJGnLfuE6O1c/Z2YQ0Y/jw== --osd-uuid 91b8d0ad-91d5-4e66-8994-4bb462627f30 2026-02-25T22:28:19.277 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:19.272+0000 7fbeca91aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:19.279 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:19.272+0000 7fbeca91aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:19.280 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:19.276+0000 7fbeca91aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:19.280 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:19.276+0000 7fbeca91aa40 -1 bdev(0x55ad2f9aa400 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:19.280 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:19.276+0000 7fbeca91aa40 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T22:28:19.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T22:28:19.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:19.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T22:28:19.912 INFO:tasks.workunit.client.0.trial125.stdout:adding osd4 key to auth repository 2026-02-25T22:28:19.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:20.199 INFO:tasks.workunit.client.0.trial125.stdout:start osd.4 2026-02-25T22:28:20.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T22:28:20.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:20.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:20.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:20.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:20.227 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:20.220+0000 7f9d563a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:20.228 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:20.224+0000 7f9d563a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:20.230 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:20.224+0000 7f9d563a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:20.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T22:28:20.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:20.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T22:28:20.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:20.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:20.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:20.417 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:20.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:20.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:20.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:28:20.613 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:20.608+0000 7f9d563a9a40 -1 Falling back to public interface 2026-02-25T22:28:20.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:21.213 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:21.208+0000 7f9d563a9a40 -1 osd.4 0 log_to_monitors true 2026-02-25T22:28:21.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:21.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:21.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:21.635 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:21.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:21.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:28:21.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:22.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:22.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:22.848 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:28:22.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:22.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:22.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:28:23.062 INFO:tasks.workunit.client.0.trial125.stdout:osd.4 up in weight 1 up_from 35 up_thru 36 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4188221709,v1:127.0.0.1:6835/4188221709] [v2:127.0.0.1:6836/4188221709,v1:127.0.0.1:6837/4188221709] exists,up 91b8d0ad-91d5-4e66-8994-4bb462627f30 2026-02-25T22:28:23.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:23.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:23.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:23.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:28:23.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-25T22:28:23.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:28:23.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:23.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T22:28:23.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:23.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:23.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:23.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:23.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:23.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:23.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:23.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-25T22:28:23.069 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:23.070 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e0cb1611-faf1-482b-a150-8c4e341a542a 2026-02-25T22:28:23.070 INFO:tasks.workunit.client.0.trial125.stdout:add osd5 e0cb1611-faf1-482b-a150-8c4e341a542a 2026-02-25T22:28:23.070 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 e0cb1611-faf1-482b-a150-8c4e341a542a' 2026-02-25T22:28:23.071 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:23.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCHd59p0n7SBBAAr4WBqi8lBc5Ng8XDiZyTnQ== 2026-02-25T22:28:23.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCHd59p0n7SBBAAr4WBqi8lBc5Ng8XDiZyTnQ=="}' 2026-02-25T22:28:23.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e0cb1611-faf1-482b-a150-8c4e341a542a -i td/test-erasure-eio/5/new.json 2026-02-25T22:28:23.296 INFO:tasks.workunit.client.0.trial125.stdout:5 2026-02-25T22:28:23.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-25T22:28:23.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHd59p0n7SBBAAr4WBqi8lBc5Ng8XDiZyTnQ== --osd-uuid e0cb1611-faf1-482b-a150-8c4e341a542a 2026-02-25T22:28:23.334 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:23.328+0000 7fecb6982a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:23.335 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:23.332+0000 7fecb6982a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:23.336 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:23.332+0000 7fecb6982a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:23.337 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:23.332+0000 7fecb6982a40 -1 bdev(0x5637d41a2400 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:23.337 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:23.332+0000 7fecb6982a40 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-25T22:28:23.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-25T22:28:23.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:24.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T22:28:24.000 INFO:tasks.workunit.client.0.trial125.stdout:adding osd5 key to auth repository 2026-02-25T22:28:24.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:24.297 INFO:tasks.workunit.client.0.trial125.stdout:start osd.5 2026-02-25T22:28:24.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T22:28:24.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:24.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:24.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:24.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:24.323 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:24.316+0000 7f85bd9f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:24.325 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:24.320+0000 7f85bd9f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:24.327 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:24.320+0000 7f85bd9f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:24.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T22:28:24.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:24.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T22:28:24.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:24.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:24.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:24.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:24.513 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:24.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:24.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:28:24.713 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:24.708+0000 7f85bd9f4a40 -1 Falling back to public interface 2026-02-25T22:28:24.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:25.362 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:25.356+0000 7f85bd9f4a40 -1 osd.5 0 log_to_monitors true 2026-02-25T22:28:25.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:25.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:25.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:25.724 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:25.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:25.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:28:25.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:26.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:26.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:26.949 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:28:26.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:26.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:26.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:28:27.161 INFO:tasks.workunit.client.0.trial125.stdout:osd.5 up in weight 1 up_from 44 up_thru 45 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/4063364292,v1:127.0.0.1:6843/4063364292] [v2:127.0.0.1:6844/4063364292,v1:127.0.0.1:6845/4063364292] exists,up e0cb1611-faf1-482b-a150-8c4e341a542a 2026-02-25T22:28:27.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:27.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:27.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:27.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:28:27.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-25T22:28:27.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:28:27.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:27.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T22:28:27.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:27.166 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:27.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:27.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:27.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:27.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:27.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:27.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-25T22:28:27.168 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:27.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2627aeda-f614-4fa6-8f35-1a70222fb540 2026-02-25T22:28:27.169 INFO:tasks.workunit.client.0.trial125.stdout:add osd6 2627aeda-f614-4fa6-8f35-1a70222fb540 2026-02-25T22:28:27.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 2627aeda-f614-4fa6-8f35-1a70222fb540' 2026-02-25T22:28:27.169 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:27.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCLd59pHFGdChAAbAyQsXtxW/WWo0RProxOhg== 2026-02-25T22:28:27.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCLd59pHFGdChAAbAyQsXtxW/WWo0RProxOhg=="}' 2026-02-25T22:28:27.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2627aeda-f614-4fa6-8f35-1a70222fb540 -i td/test-erasure-eio/6/new.json 2026-02-25T22:28:27.390 INFO:tasks.workunit.client.0.trial125.stdout:6 2026-02-25T22:28:27.401 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-25T22:28:27.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLd59pHFGdChAAbAyQsXtxW/WWo0RProxOhg== --osd-uuid 2627aeda-f614-4fa6-8f35-1a70222fb540 2026-02-25T22:28:27.420 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:27.416+0000 7f09505aaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:27.422 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:27.416+0000 7f09505aaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:27.423 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:27.416+0000 7f09505aaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:27.423 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:27.420+0000 7f09505aaa40 -1 bdev(0x556c613d6400 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:27.423 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:27.420+0000 7f09505aaa40 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-25T22:28:28.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-25T22:28:28.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:28.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T22:28:28.048 INFO:tasks.workunit.client.0.trial125.stdout:adding osd6 key to auth repository 2026-02-25T22:28:28.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:28.339 INFO:tasks.workunit.client.0.trial125.stdout:start osd.6 2026-02-25T22:28:28.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T22:28:28.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:28.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:28.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:28.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:28.364 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:28.360+0000 7f7f682a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:28.366 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:28.360+0000 7f7f682a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:28.367 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:28.364+0000 7f7f682a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:28.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T22:28:28.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:28.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T22:28:28.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:28.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:28.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:28.551 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:28.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:28.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:28.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:28:28.748 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:28.744+0000 7f7f682a0a40 -1 Falling back to public interface 2026-02-25T22:28:28.766 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:29.406 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:29.400+0000 7f7f682a0a40 -1 osd.6 0 log_to_monitors true 2026-02-25T22:28:29.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:29.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:29.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:29.768 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:29.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:29.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:28:29.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:30.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:30.975 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:28:30.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:30.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:30.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:30.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:28:31.184 INFO:tasks.workunit.client.0.trial125.stdout:osd.6 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3312262879,v1:127.0.0.1:6851/3312262879] [v2:127.0.0.1:6852/3312262879,v1:127.0.0.1:6853/3312262879] exists,up 2627aeda-f614-4fa6-8f35-1a70222fb540 2026-02-25T22:28:31.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:31.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:31.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:31.186 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:28:31.186 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:28:31.186 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:28:31.187 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:31.187 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:31.187 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:31.187 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:28:31.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:28:31.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:28:31.248 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:28:31.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:28:31.256 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:28:05.748+0000 7f02d7ed0a40 0 load: jerasure load: lrc 2026-02-25T22:28:31.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:477: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:28:31.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:479: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2026-02-25T22:28:31.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T22:28:31.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:28:31.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:28:31.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T22:28:31.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:28:31.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T22:28:31.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:28:31.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T22:28:31.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:28:31.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:28:31.901 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:28:31.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:28:32.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:28:32.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:28:32.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:28:32.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:28:32.912 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:28:32.913 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:28:32.913 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:28:32.914 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:28:32.914 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:28:32.914 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:28:33.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:28:33.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:28:33.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:28:33.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:28:33.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:28:33.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:28:33.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:28:33.239 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:28:33.239 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:28:33.239 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:28:33.239 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:28:33.239 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:28:33.239 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:28:33.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:28:33.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:33.240 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:28:33.317 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803785 2026-02-25T22:28:33.318 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803785 2026-02-25T22:28:33.318 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785' 2026-02-25T22:28:33.318 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:33.318 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:28:33.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574855 2026-02-25T22:28:33.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574855 2026-02-25T22:28:33.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574855' 2026-02-25T22:28:33.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:33.397 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:28:33.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378630 2026-02-25T22:28:33.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378630 2026-02-25T22:28:33.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574855 2-81604378630' 2026-02-25T22:28:33.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:33.473 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:28:33.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149701 2026-02-25T22:28:33.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149701 2026-02-25T22:28:33.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574855 2-81604378630 3-111669149701' 2026-02-25T22:28:33.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:33.552 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:28:33.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855365 2026-02-25T22:28:33.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855365 2026-02-25T22:28:33.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574855 2-81604378630 3-111669149701 4-150323855365' 2026-02-25T22:28:33.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:33.629 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:28:33.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561028 2026-02-25T22:28:33.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561028 2026-02-25T22:28:33.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574855 2-81604378630 3-111669149701 4-150323855365 5-188978561028' 2026-02-25T22:28:33.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:33.708 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:28:33.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=223338299395 2026-02-25T22:28:33.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 223338299395 2026-02-25T22:28:33.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574855 2-81604378630 3-111669149701 4-150323855365 5-188978561028 6-223338299395' 2026-02-25T22:28:33.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:33.791 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803785 2026-02-25T22:28:33.791 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:33.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:28:33.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803785 2026-02-25T22:28:33.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:33.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803785 2026-02-25T22:28:33.794 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803785 2026-02-25T22:28:33.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2026-02-25T22:28:33.795 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:28:34.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803783 -lt 25769803785 2026-02-25T22:28:34.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:28:35.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:28:35.003 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:28:35.218 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803783 -lt 25769803785 2026-02-25T22:28:35.218 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:28:36.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:28:36.220 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:28:36.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803785 2026-02-25T22:28:36.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:36.436 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574855 2026-02-25T22:28:36.436 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:36.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:28:36.438 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574855 2026-02-25T22:28:36.438 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:36.440 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574855 2026-02-25T22:28:36.440 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574855' 2026-02-25T22:28:36.440 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574855 2026-02-25T22:28:36.441 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:28:36.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574855 -lt 55834574855 2026-02-25T22:28:36.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:36.654 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378630 2026-02-25T22:28:36.654 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:36.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:28:36.655 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:36.656 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378630 2026-02-25T22:28:36.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378630 2026-02-25T22:28:36.657 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 81604378630 2026-02-25T22:28:36.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378630' 2026-02-25T22:28:36.657 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:28:36.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378630 -lt 81604378630 2026-02-25T22:28:36.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:36.863 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149701 2026-02-25T22:28:36.863 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:36.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:28:36.865 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149701 2026-02-25T22:28:36.865 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:36.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149701 2026-02-25T22:28:36.867 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149701 2026-02-25T22:28:36.867 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149701' 2026-02-25T22:28:36.867 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:28:37.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149702 -lt 111669149701 2026-02-25T22:28:37.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:37.086 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855365 2026-02-25T22:28:37.086 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:37.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:28:37.088 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855365 2026-02-25T22:28:37.088 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:37.089 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855365 2026-02-25T22:28:37.089 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855365' 2026-02-25T22:28:37.090 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 150323855365 2026-02-25T22:28:37.090 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:28:37.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855365 -lt 150323855365 2026-02-25T22:28:37.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:37.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-188978561028 2026-02-25T22:28:37.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:37.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:28:37.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-188978561028 2026-02-25T22:28:37.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:37.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561028 2026-02-25T22:28:37.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 188978561028' 2026-02-25T22:28:37.312 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 188978561028 2026-02-25T22:28:37.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:28:37.517 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561029 -lt 188978561028 2026-02-25T22:28:37.517 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:37.518 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-223338299395 2026-02-25T22:28:37.518 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:37.519 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:28:37.520 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-223338299395 2026-02-25T22:28:37.520 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:37.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=223338299395 2026-02-25T22:28:37.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 223338299395' 2026-02-25T22:28:37.522 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 223338299395 2026-02-25T22:28:37.522 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:28:37.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 223338299396 -lt 223338299395 2026-02-25T22:28:37.740 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:28:37.741 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:28:37.741 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:28:38.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:28:38.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:28:38.019 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:28:38.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:28:38.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:28:38.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:28:38.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:28:38.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:28:38.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:28:38.227 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:28:38.228 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:28:38.228 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:28:38.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:28:38.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:482: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:28:38.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.517 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:28:38.517 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2026-02-25T22:28:38.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2026-02-25T22:28:38.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:28:38.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:28:38.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2026-02-25T22:28:38.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:28:38.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:28:38.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:28:38.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:28:38.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2026-02-25T22:28:38.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:28:38.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2026-02-25T22:28:38.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects: ceph osd out 0 2026-02-25T22:28:38.841 INFO:tasks.workunit.client.0.trial125.stderr:osd.0 is already out. 2026-02-25T22:28:38.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects: wait_for_clean 2026-02-25T22:28:38.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:28:38.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:28:38.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:28:38.851 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:28:38.852 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:28:38.852 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:28:38.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:28:38.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:28:38.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:28:38.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:28:38.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:28:38.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:28:38.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:28:38.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:28:38.958 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:28:39.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:28:39.160 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:28:39.160 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:28:39.160 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:28:39.160 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:28:39.161 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:28:39.161 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:28:39.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:28:39.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:39.161 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:28:39.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803788 2026-02-25T22:28:39.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803788 2026-02-25T22:28:39.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788' 2026-02-25T22:28:39.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:39.240 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:28:39.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574858 2026-02-25T22:28:39.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574858 2026-02-25T22:28:39.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-55834574858' 2026-02-25T22:28:39.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:39.320 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:28:39.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378633 2026-02-25T22:28:39.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378633 2026-02-25T22:28:39.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-55834574858 2-81604378633' 2026-02-25T22:28:39.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:39.398 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:28:39.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149704 2026-02-25T22:28:39.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149704 2026-02-25T22:28:39.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-55834574858 2-81604378633 3-111669149704' 2026-02-25T22:28:39.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:39.477 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:28:39.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855368 2026-02-25T22:28:39.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855368 2026-02-25T22:28:39.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-55834574858 2-81604378633 3-111669149704 4-150323855368' 2026-02-25T22:28:39.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:39.556 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:28:39.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561031 2026-02-25T22:28:39.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561031 2026-02-25T22:28:39.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-55834574858 2-81604378633 3-111669149704 4-150323855368 5-188978561031' 2026-02-25T22:28:39.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:39.634 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:28:39.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=223338299398 2026-02-25T22:28:39.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 223338299398 2026-02-25T22:28:39.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-55834574858 2-81604378633 3-111669149704 4-150323855368 5-188978561031 6-223338299398' 2026-02-25T22:28:39.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:39.715 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803788 2026-02-25T22:28:39.715 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:39.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:28:39.717 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803788 2026-02-25T22:28:39.717 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:39.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803788 2026-02-25T22:28:39.718 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803788 2026-02-25T22:28:39.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803788' 2026-02-25T22:28:39.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:28:39.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803788 -lt 25769803788 2026-02-25T22:28:39.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:39.938 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574858 2026-02-25T22:28:39.938 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:39.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:28:39.940 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574858 2026-02-25T22:28:39.940 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:39.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574858 2026-02-25T22:28:39.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574858' 2026-02-25T22:28:39.942 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574858 2026-02-25T22:28:39.942 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:28:40.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574856 -lt 55834574858 2026-02-25T22:28:40.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:28:41.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:28:41.159 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:28:41.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574858 -lt 55834574858 2026-02-25T22:28:41.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:41.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378633 2026-02-25T22:28:41.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:41.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:28:41.377 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378633 2026-02-25T22:28:41.377 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:41.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378633 2026-02-25T22:28:41.379 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 81604378633 2026-02-25T22:28:41.379 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378633' 2026-02-25T22:28:41.379 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:28:41.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378634 -lt 81604378633 2026-02-25T22:28:41.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:41.597 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149704 2026-02-25T22:28:41.597 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:41.598 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:28:41.599 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149704 2026-02-25T22:28:41.599 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:41.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149704 2026-02-25T22:28:41.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149704' 2026-02-25T22:28:41.601 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149704 2026-02-25T22:28:41.602 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:28:41.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149705 -lt 111669149704 2026-02-25T22:28:41.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:41.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855368 2026-02-25T22:28:41.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:41.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:28:41.825 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855368 2026-02-25T22:28:41.825 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:41.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855368 2026-02-25T22:28:41.827 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 150323855368 2026-02-25T22:28:41.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855368' 2026-02-25T22:28:41.827 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:28:42.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855368 -lt 150323855368 2026-02-25T22:28:42.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:42.052 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-188978561031 2026-02-25T22:28:42.052 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:42.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:28:42.054 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-188978561031 2026-02-25T22:28:42.055 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:42.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561031 2026-02-25T22:28:42.056 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 188978561031 2026-02-25T22:28:42.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 188978561031' 2026-02-25T22:28:42.057 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:28:42.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561031 -lt 188978561031 2026-02-25T22:28:42.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:42.267 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-223338299398 2026-02-25T22:28:42.267 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:42.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:28:42.269 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-223338299398 2026-02-25T22:28:42.269 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:42.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=223338299398 2026-02-25T22:28:42.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 223338299398' 2026-02-25T22:28:42.271 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 223338299398 2026-02-25T22:28:42.271 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:28:42.484 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 223338299399 -lt 223338299398 2026-02-25T22:28:42.484 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:28:42.485 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:28:42.485 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:28:42.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:28:42.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:28:42.767 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:28:42.767 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:28:42.767 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:28:42.767 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:28:42.768 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:28:42.768 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:28:42.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:28:42.976 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:28:42.977 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:28:42.977 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:28:43.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:28:43.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:28:43.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:28:43.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2026-02-25T22:28:43.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:28:43.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:28:43.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2026-02-25T22:28:43.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:28:43.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:28:43.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2026-02-25T22:28:43.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:28:43.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:28:43.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2026-02-25T22:28:43.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:28:43.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:28:43.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2026-02-25T22:28:43.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:28:43.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:28:43.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2026-02-25T22:28:43.302 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:28:43.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:28:43.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2026-02-25T22:28:43.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:28:43.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:28:43.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2026-02-25T22:28:43.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:28:43.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:28:43.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2026-02-25T22:28:43.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:28:43.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:28:43.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:495: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:28:43.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:28:43.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:28:43.592 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:28:43.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:28:43.875 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:28:43.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:28:43.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:28:43.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:43.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:28:43.886 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:43.886 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:43.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:43.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:43.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:43.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:43.998 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:43.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:43.999 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:44.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:44.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:44.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:44.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:44.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:44.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:44.003 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:44.004 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:44.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:44.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:28:44.027 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:44.027 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:44.027 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:44.027 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:44.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:44.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:44.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:44.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:44.032 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:44.033 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:44.033 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:44.034 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:44.034 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:44.034 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:44.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:44.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:44.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:44.036 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:44.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:44.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:44.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:28:44.040 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:44.040 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:44.040 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:44.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:28:44.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:44.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:44.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:28:44.043 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:44.043 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:44.043 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:44.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:28:44.045 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:44.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:28:44.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:44.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:28:44.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:28:44.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:28:44.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:28:44.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:28:44.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:28:44.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:28:44.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:28:44.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:28:44.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:44.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:44.068 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:44.068 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:44.068 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:44.068 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:44.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:28:44.098 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:28:44.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:28:44.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:44.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:44.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:28:44.100 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:28:44.100 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:44.100 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:44.100 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:44.100 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:44.101 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:44.101 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:44.101 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:28:44.101 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:44.101 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:28:44.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:28:44.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:44.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:44.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:28:44.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:28:44.157 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:44.157 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:44.157 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:44.157 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:44.157 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:44.157 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:44.157 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:28:44.158 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:44.158 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:28:44.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:28:44.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:28:44.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:28:44.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:28:44.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:28:44.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:28:44.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:28:44.324 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:28:44.324 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:44.324 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:44.324 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:44.324 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:44.324 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:44.324 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:44.325 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:44.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:44.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:28:44.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:28:44.450 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:28:44.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:28:45.461 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:28:45.461 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:45.461 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:45.461 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:45.462 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:45.462 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:45.462 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:28:45.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:28:45.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:28:45.511 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:28:45.517 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:28:45.518 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:28:44.083+0000 7f8f4ca11ec0 0 load: jerasure load: lrc 2026-02-25T22:28:45.519 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2026-02-25T22:28:45.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2026-02-25T22:28:45.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:501: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2026-02-25T22:28:45.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:28:45.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:45.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:505: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2026-02-25T22:28:45.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-25T22:28:45.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:28:45.520 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-25T22:28:45.521 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-25T22:28:45.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:28:45.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:28:45.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:28:45.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:45.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:28:45.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:45.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:28:45.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:45.525 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:45.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:45.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:45.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:45.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:28:45.527 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:45.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0997b71c-47b4-42ed-b47a-166fe2f243ef 2026-02-25T22:28:45.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0997b71c-47b4-42ed-b47a-166fe2f243ef' 2026-02-25T22:28:45.528 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 0997b71c-47b4-42ed-b47a-166fe2f243ef 2026-02-25T22:28:45.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:45.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCdd59pFC9aIBAAYU2jf1eD+PJb1KdnqMoG8g== 2026-02-25T22:28:45.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCdd59pFC9aIBAAYU2jf1eD+PJb1KdnqMoG8g=="}' 2026-02-25T22:28:45.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0997b71c-47b4-42ed-b47a-166fe2f243ef -i td/test-erasure-eio/0/new.json 2026-02-25T22:28:45.656 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:45.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:28:45.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdd59pFC9aIBAAYU2jf1eD+PJb1KdnqMoG8g== --osd-uuid 0997b71c-47b4-42ed-b47a-166fe2f243ef 2026-02-25T22:28:45.683 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:45.675+0000 7f4efb99da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:45.685 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:45.679+0000 7f4efb99da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:45.686 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:45.679+0000 7f4efb99da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:45.686 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:45.679+0000 7f4efb99da40 -1 bdev(0x5609a68e6400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:45.687 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:45.679+0000 7f4efb99da40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:28:46.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:28:46.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:46.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:28:46.358 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:28:46.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:46.498 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:28:46.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:28:46.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:46.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:46.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:46.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:46.525 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:46.519+0000 7fb9981ada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:46.526 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:46.519+0000 7fb9981ada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:46.527 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:46.523+0000 7fb9981ada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:46.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:28:46.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:46.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:46.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:46.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:46.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:46.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:46.627 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:46.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:46.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:46.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:46.916 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:46.911+0000 7fb9981ada40 -1 Falling back to public interface 2026-02-25T22:28:47.461 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:47.455+0000 7fb9981ada40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:28:47.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:47.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:47.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:47.845 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:47.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:47.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:48.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:49.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:49.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:49.059 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:28:49.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:49.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:49.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:49.268 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2092435643,v1:127.0.0.1:6803/2092435643] [v2:127.0.0.1:6804/2092435643,v1:127.0.0.1:6805/2092435643] exists,up 0997b71c-47b4-42ed-b47a-166fe2f243ef 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:28:49.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:49.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:49.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:28:49.270 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:49.270 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:49.270 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:49.270 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:49.270 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:49.270 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:49.270 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:49.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:28:49.272 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:49.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=71e5689f-59d1-42d9-a4ed-99f4fc895acf 2026-02-25T22:28:49.273 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 71e5689f-59d1-42d9-a4ed-99f4fc895acf 2026-02-25T22:28:49.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 71e5689f-59d1-42d9-a4ed-99f4fc895acf' 2026-02-25T22:28:49.273 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:49.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQChd59puEAIERAA/11UCXs/O2Rk82itdcXSuQ== 2026-02-25T22:28:49.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQChd59puEAIERAA/11UCXs/O2Rk82itdcXSuQ=="}' 2026-02-25T22:28:49.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 71e5689f-59d1-42d9-a4ed-99f4fc895acf -i td/test-erasure-eio/1/new.json 2026-02-25T22:28:49.501 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:49.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:28:49.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChd59puEAIERAA/11UCXs/O2Rk82itdcXSuQ== --osd-uuid 71e5689f-59d1-42d9-a4ed-99f4fc895acf 2026-02-25T22:28:49.538 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:49.531+0000 7f26b8c1ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:49.540 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:49.535+0000 7f26b8c1ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:49.541 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:49.535+0000 7f26b8c1ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:49.542 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:49.535+0000 7f26b8c1ca40 -1 bdev(0x558c938a0400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:49.542 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:49.535+0000 7f26b8c1ca40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:28:50.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:28:50.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:50.228 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:28:50.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:28:50.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:50.515 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:28:50.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:28:50.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:50.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:50.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:50.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:50.532 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:50.527+0000 7f32fbc62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:50.534 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:50.527+0000 7f32fbc62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:50.535 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:50.527+0000 7f32fbc62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:50.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:28:50.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:50.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:28:50.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:50.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:50.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:50.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:50.729 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:50.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:50.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:28:50.917 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:50.911+0000 7f32fbc62a40 -1 Falling back to public interface 2026-02-25T22:28:50.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:51.521 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:51.515+0000 7f32fbc62a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:28:51.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:51.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:51.944 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:51.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:51.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:51.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:28:52.151 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/35180204,v1:127.0.0.1:6811/35180204] [v2:127.0.0.1:6812/35180204,v1:127.0.0.1:6813/35180204] exists,up 71e5689f-59d1-42d9-a4ed-99f4fc895acf 2026-02-25T22:28:52.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:52.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:52.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:52.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:28:52.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:28:52.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:28:52.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:52.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:28:52.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:52.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:28:52.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:52.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:52.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:52.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:52.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:52.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:52.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:52.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:52.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:28:52.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:28:52.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:52.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:52.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:52.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:52.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:52.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:52.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:28:52.158 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:52.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c3056cf4-d9be-4505-abb0-307c714c06ec 2026-02-25T22:28:52.158 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 c3056cf4-d9be-4505-abb0-307c714c06ec 2026-02-25T22:28:52.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c3056cf4-d9be-4505-abb0-307c714c06ec' 2026-02-25T22:28:52.159 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:52.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCkd59pdD4PChAAXwCPjot+5IUUVan5ZkMlGQ== 2026-02-25T22:28:52.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCkd59pdD4PChAAXwCPjot+5IUUVan5ZkMlGQ=="}' 2026-02-25T22:28:52.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c3056cf4-d9be-4505-abb0-307c714c06ec -i td/test-erasure-eio/2/new.json 2026-02-25T22:28:52.391 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:28:52.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:28:52.404 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCkd59pdD4PChAAXwCPjot+5IUUVan5ZkMlGQ== --osd-uuid c3056cf4-d9be-4505-abb0-307c714c06ec 2026-02-25T22:28:52.427 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:52.423+0000 7f40130b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:52.429 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:52.423+0000 7f40130b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:52.430 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:52.423+0000 7f40130b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:52.431 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:52.423+0000 7f40130b6a40 -1 bdev(0x559ac4b6e400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:52.431 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:52.423+0000 7f40130b6a40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:28:53.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:28:53.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:53.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:28:53.084 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:28:53.084 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:53.368 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:28:53.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:28:53.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:53.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:53.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:53.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:53.393 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:53.387+0000 7fe7c45d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:53.395 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:53.391+0000 7fe7c45d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:53.396 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:53.391+0000 7fe7c45d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:53.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:28:53.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:53.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:28:53.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:53.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:53.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:53.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:53.582 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:53.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:53.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:28:53.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:53.805 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:53.799+0000 7fe7c45d7a40 -1 Falling back to public interface 2026-02-25T22:28:54.405 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:54.399+0000 7fe7c45d7a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:28:54.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:54.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:54.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:54.792 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:54.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:54.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:28:54.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:56.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:56.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:56.002 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:28:56.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:56.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:56.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:28:56.208 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 18 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4029450051,v1:127.0.0.1:6819/4029450051] [v2:127.0.0.1:6820/4029450051,v1:127.0.0.1:6821/4029450051] exists,up c3056cf4-d9be-4505-abb0-307c714c06ec 2026-02-25T22:28:56.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:56.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:56.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:56.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:28:56.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:28:56.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:56.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:56.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:56.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:28:56.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:28:56.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:56.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:56.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:56.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:56.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:56.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:56.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:28:56.214 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:56.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=85344e52-9ddc-435d-8b8c-cd340501dc18 2026-02-25T22:28:56.215 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 85344e52-9ddc-435d-8b8c-cd340501dc18 2026-02-25T22:28:56.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 85344e52-9ddc-435d-8b8c-cd340501dc18' 2026-02-25T22:28:56.216 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:56.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCod59pms1sDRAAsTVdJJ3cG99N8F7AA05tlw== 2026-02-25T22:28:56.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCod59pms1sDRAAsTVdJJ3cG99N8F7AA05tlw=="}' 2026-02-25T22:28:56.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 85344e52-9ddc-435d-8b8c-cd340501dc18 -i td/test-erasure-eio/3/new.json 2026-02-25T22:28:56.434 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:28:56.443 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:28:56.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCod59pms1sDRAAsTVdJJ3cG99N8F7AA05tlw== --osd-uuid 85344e52-9ddc-435d-8b8c-cd340501dc18 2026-02-25T22:28:56.470 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:56.463+0000 7fd02638da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:56.472 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:56.467+0000 7fd02638da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:56.473 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:56.467+0000 7fd02638da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:56.473 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:56.467+0000 7fd02638da40 -1 bdev(0x564766f7e400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:56.474 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:56.467+0000 7fd02638da40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:28:57.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:28:57.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:57.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:28:57.172 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:28:57.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:57.457 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:28:57.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:28:57.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:57.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:57.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:57.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:57.483 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:57.479+0000 7f2104190a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:57.485 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:57.479+0000 7f2104190a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:57.486 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:57.479+0000 7f2104190a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:57.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:28:57.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:57.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:28:57.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:57.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:57.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:57.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:57.674 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:28:57.674 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:57.675 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:28:57.849 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:57.843+0000 7f2104190a40 -1 Falling back to public interface 2026-02-25T22:28:57.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:58.506 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:28:58.499+0000 7f2104190a40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:28:58.883 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:58.883 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:58.883 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:28:58.884 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:58.884 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:58.884 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:28:59.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:00.098 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:00.098 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:00.098 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:00.098 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:29:00.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:00.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:29:00.306 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 26 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/105868637,v1:127.0.0.1:6827/105868637] [v2:127.0.0.1:6828/105868637,v1:127.0.0.1:6829/105868637] exists,up 85344e52-9ddc-435d-8b8c-cd340501dc18 2026-02-25T22:29:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:29:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T22:29:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:29:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:00.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:29:00.311 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:29:00.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:00.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:00.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:00.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:00.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:00.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:00.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T22:29:00.314 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:00.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5d5032d3-f8b9-41ae-aae7-6999b2485945 2026-02-25T22:29:00.314 INFO:tasks.workunit.client.0.trial125.stdout:add osd4 5d5032d3-f8b9-41ae-aae7-6999b2485945 2026-02-25T22:29:00.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 5d5032d3-f8b9-41ae-aae7-6999b2485945' 2026-02-25T22:29:00.315 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:00.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCsd59pcJ5aExAA0OI6EqogOQrqDWvo2hVKUQ== 2026-02-25T22:29:00.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCsd59pcJ5aExAA0OI6EqogOQrqDWvo2hVKUQ=="}' 2026-02-25T22:29:00.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5d5032d3-f8b9-41ae-aae7-6999b2485945 -i td/test-erasure-eio/4/new.json 2026-02-25T22:29:00.548 INFO:tasks.workunit.client.0.trial125.stdout:4 2026-02-25T22:29:00.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T22:29:00.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCsd59pcJ5aExAA0OI6EqogOQrqDWvo2hVKUQ== --osd-uuid 5d5032d3-f8b9-41ae-aae7-6999b2485945 2026-02-25T22:29:00.585 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:00.579+0000 7f587e3fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:00.587 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:00.579+0000 7f587e3fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:00.588 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:00.583+0000 7f587e3fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:00.588 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:00.583+0000 7f587e3fca40 -1 bdev(0x558b7d198400 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:00.588 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:00.583+0000 7f587e3fca40 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T22:29:01.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T22:29:01.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:01.251 INFO:tasks.workunit.client.0.trial125.stdout:adding osd4 key to auth repository 2026-02-25T22:29:01.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T22:29:01.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:01.547 INFO:tasks.workunit.client.0.trial125.stdout:start osd.4 2026-02-25T22:29:01.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T22:29:01.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:01.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:01.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:01.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:01.564 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:01.559+0000 7fd798be3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:01.565 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:01.559+0000 7fd798be3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:01.566 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:01.559+0000 7fd798be3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:01.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T22:29:01.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:01.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T22:29:01.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:01.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:01.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:01.762 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:29:01.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:01.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:01.763 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:29:01.968 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:02.009 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:02.003+0000 7fd798be3a40 -1 Falling back to public interface 2026-02-25T22:29:02.653 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:02.647+0000 7fd798be3a40 -1 osd.4 0 log_to_monitors true 2026-02-25T22:29:02.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:02.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:02.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:02.970 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:29:02.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:02.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:29:03.189 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:04.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:04.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:04.191 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:29:04.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:04.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:04.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:29:04.402 INFO:tasks.workunit.client.0.trial125.stdout:osd.4 up in weight 1 up_from 34 up_thru 35 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2704750277,v1:127.0.0.1:6835/2704750277] [v2:127.0.0.1:6836/2704750277,v1:127.0.0.1:6837/2704750277] exists,up 5d5032d3-f8b9-41ae-aae7-6999b2485945 2026-02-25T22:29:04.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:04.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:04.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:04.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:29:04.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-25T22:29:04.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:29:04.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:04.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T22:29:04.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:04.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-25T22:29:04.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:29:04.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:04.407 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:04.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:04.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:04.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-25T22:29:04.409 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:04.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9969d7ad-c26f-43ee-8697-3a03775b2e5b 2026-02-25T22:29:04.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 9969d7ad-c26f-43ee-8697-3a03775b2e5b' 2026-02-25T22:29:04.410 INFO:tasks.workunit.client.0.trial125.stdout:add osd5 9969d7ad-c26f-43ee-8697-3a03775b2e5b 2026-02-25T22:29:04.410 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:04.420 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCwd59p63H8GBAAOWXh0B05Q/XW64TSFoQuOg== 2026-02-25T22:29:04.420 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCwd59p63H8GBAAOWXh0B05Q/XW64TSFoQuOg=="}' 2026-02-25T22:29:04.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9969d7ad-c26f-43ee-8697-3a03775b2e5b -i td/test-erasure-eio/5/new.json 2026-02-25T22:29:04.638 INFO:tasks.workunit.client.0.trial125.stdout:5 2026-02-25T22:29:04.648 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-25T22:29:04.649 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwd59p63H8GBAAOWXh0B05Q/XW64TSFoQuOg== --osd-uuid 9969d7ad-c26f-43ee-8697-3a03775b2e5b 2026-02-25T22:29:04.666 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:04.659+0000 7f0ac1df7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:04.668 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:04.663+0000 7f0ac1df7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:04.669 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:04.663+0000 7f0ac1df7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:04.669 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:04.663+0000 7f0ac1df7a40 -1 bdev(0x558cee87e400 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:04.669 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:04.663+0000 7f0ac1df7a40 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-25T22:29:05.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-25T22:29:05.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:05.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T22:29:05.299 INFO:tasks.workunit.client.0.trial125.stdout:adding osd5 key to auth repository 2026-02-25T22:29:05.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:05.594 INFO:tasks.workunit.client.0.trial125.stdout:start osd.5 2026-02-25T22:29:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T22:29:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:05.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:05.621 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:05.615+0000 7feaa6bf0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:05.622 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:05.615+0000 7feaa6bf0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:05.623 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:05.619+0000 7feaa6bf0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:05.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T22:29:05.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:05.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T22:29:05.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:05.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:05.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:05.813 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:29:05.813 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:05.813 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:05.813 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:29:06.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:06.077 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:06.071+0000 7feaa6bf0a40 -1 Falling back to public interface 2026-02-25T22:29:06.670 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:06.663+0000 7feaa6bf0a40 -1 osd.5 0 log_to_monitors true 2026-02-25T22:29:07.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:07.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:07.024 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:29:07.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:07.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:07.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:29:07.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:08.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:08.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:08.240 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:29:08.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:08.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:08.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:29:08.451 INFO:tasks.workunit.client.0.trial125.stdout:osd.5 up in weight 1 up_from 42 up_thru 43 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/233163345,v1:127.0.0.1:6843/233163345] [v2:127.0.0.1:6844/233163345,v1:127.0.0.1:6845/233163345] exists,up 9969d7ad-c26f-43ee-8697-3a03775b2e5b 2026-02-25T22:29:08.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:08.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:08.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:08.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:29:08.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-25T22:29:08.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:29:08.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:08.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T22:29:08.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:08.455 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:08.456 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:08.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:08.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:08.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:08.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:08.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-25T22:29:08.458 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:08.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=60ce7625-cd9e-4793-a6e5-a2e0916279b6 2026-02-25T22:29:08.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 60ce7625-cd9e-4793-a6e5-a2e0916279b6' 2026-02-25T22:29:08.459 INFO:tasks.workunit.client.0.trial125.stdout:add osd6 60ce7625-cd9e-4793-a6e5-a2e0916279b6 2026-02-25T22:29:08.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:08.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC0d59pRc3gGxAA22POEUxUBGCAW2FBRir++Q== 2026-02-25T22:29:08.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC0d59pRc3gGxAA22POEUxUBGCAW2FBRir++Q=="}' 2026-02-25T22:29:08.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 60ce7625-cd9e-4793-a6e5-a2e0916279b6 -i td/test-erasure-eio/6/new.json 2026-02-25T22:29:08.689 INFO:tasks.workunit.client.0.trial125.stdout:6 2026-02-25T22:29:08.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-25T22:29:08.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0d59pRc3gGxAA22POEUxUBGCAW2FBRir++Q== --osd-uuid 60ce7625-cd9e-4793-a6e5-a2e0916279b6 2026-02-25T22:29:08.718 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:08.711+0000 7fadfbf78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:08.719 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:08.715+0000 7fadfbf78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:08.720 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:08.715+0000 7fadfbf78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:08.721 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:08.715+0000 7fadfbf78a40 -1 bdev(0x55be0de5a400 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:08.721 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:08.715+0000 7fadfbf78a40 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-25T22:29:09.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-25T22:29:09.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:09.343 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T22:29:09.343 INFO:tasks.workunit.client.0.trial125.stdout:adding osd6 key to auth repository 2026-02-25T22:29:09.343 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:09.640 INFO:tasks.workunit.client.0.trial125.stdout:start osd.6 2026-02-25T22:29:09.640 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T22:29:09.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:09.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:09.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:09.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:09.665 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:09.659+0000 7f5455eb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:09.667 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:09.659+0000 7f5455eb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:09.668 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:09.663+0000 7f5455eb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:09.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T22:29:09.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:09.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T22:29:09.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:09.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:09.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:09.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:09.849 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:29:09.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:09.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:29:10.057 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:10.051+0000 7f5455eb7a40 -1 Falling back to public interface 2026-02-25T22:29:10.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:10.634 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:10.627+0000 7f5455eb7a40 -1 osd.6 0 log_to_monitors true 2026-02-25T22:29:11.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:11.067 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:29:11.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:11.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:11.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:11.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:29:11.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:12.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:12.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:12.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:12.299 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:29:12.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:12.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:29:12.510 INFO:tasks.workunit.client.0.trial125.stdout:osd.6 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3843884906,v1:127.0.0.1:6851/3843884906] [v2:127.0.0.1:6852/3843884906,v1:127.0.0.1:6853/3843884906] exists,up 60ce7625-cd9e-4793-a6e5-a2e0916279b6 2026-02-25T22:29:12.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:12.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:12.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:12.512 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:29:12.512 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:29:12.512 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:29:12.512 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:12.512 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:12.512 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:12.513 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:29:12.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:29:12.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:29:12.574 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:29:12.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:29:12.583 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:28:46.911+0000 7fb9981ada40 0 load: jerasure load: lrc 2026-02-25T22:29:12.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:29:12.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:508: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2026-02-25T22:29:12.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:509: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T22:29:12.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:29:12.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:29:12.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T22:29:12.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:29:12.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T22:29:12.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:29:12.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T22:29:12.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:29:12.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:29:13.215 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:29:13.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:29:14.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:29:14.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:29:14.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:29:14.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:29:14.229 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:29:14.229 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:29:14.229 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:29:14.229 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:29:14.229 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:29:14.229 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:29:14.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:29:14.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:29:14.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:29:14.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:29:14.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:29:14.333 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:29:14.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:29:14.550 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:29:14.550 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:29:14.551 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:29:14.551 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:29:14.551 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:29:14.551 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:29:14.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:29:14.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:14.551 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:29:14.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803784 2026-02-25T22:29:14.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803784 2026-02-25T22:29:14.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784' 2026-02-25T22:29:14.633 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:14.633 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:29:14.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607559 2026-02-25T22:29:14.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607559 2026-02-25T22:29:14.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559' 2026-02-25T22:29:14.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:14.713 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:29:14.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411334 2026-02-25T22:29:14.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411334 2026-02-25T22:29:14.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559 2-77309411334' 2026-02-25T22:29:14.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:14.790 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:29:14.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149702 2026-02-25T22:29:14.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149702 2026-02-25T22:29:14.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559 2-77309411334 3-111669149702' 2026-02-25T22:29:14.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:14.869 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:29:14.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888069 2026-02-25T22:29:14.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888069 2026-02-25T22:29:14.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559 2-77309411334 3-111669149702 4-146028888069' 2026-02-25T22:29:14.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:14.949 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:29:15.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626436 2026-02-25T22:29:15.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626436 2026-02-25T22:29:15.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559 2-77309411334 3-111669149702 4-146028888069 5-180388626436' 2026-02-25T22:29:15.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:15.028 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:29:15.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364803 2026-02-25T22:29:15.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364803 2026-02-25T22:29:15.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559 2-77309411334 3-111669149702 4-146028888069 5-180388626436 6-214748364803' 2026-02-25T22:29:15.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:15.108 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803784 2026-02-25T22:29:15.108 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:15.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:29:15.110 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803784 2026-02-25T22:29:15.110 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:15.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803784 2026-02-25T22:29:15.112 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803784 2026-02-25T22:29:15.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-25T22:29:15.112 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:15.323 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803784 2026-02-25T22:29:15.323 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:29:16.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:29:16.326 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:16.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803784 2026-02-25T22:29:16.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:29:17.538 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:29:17.539 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:17.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803784 -lt 25769803784 2026-02-25T22:29:17.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:17.757 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607559 2026-02-25T22:29:17.757 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:17.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:29:17.759 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607559 2026-02-25T22:29:17.759 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:17.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607559 2026-02-25T22:29:17.761 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 51539607559 2026-02-25T22:29:17.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607559' 2026-02-25T22:29:17.762 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:29:17.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607559 -lt 51539607559 2026-02-25T22:29:17.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:17.974 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411334 2026-02-25T22:29:17.975 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:17.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:29:17.976 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411334 2026-02-25T22:29:17.976 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:17.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411334 2026-02-25T22:29:17.978 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 77309411334 2026-02-25T22:29:17.979 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411334' 2026-02-25T22:29:17.979 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:29:18.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411335 -lt 77309411334 2026-02-25T22:29:18.192 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:18.193 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149702 2026-02-25T22:29:18.193 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:18.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:29:18.194 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149702 2026-02-25T22:29:18.195 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:18.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149702 2026-02-25T22:29:18.196 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149702 2026-02-25T22:29:18.197 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149702' 2026-02-25T22:29:18.197 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:29:18.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149702 -lt 111669149702 2026-02-25T22:29:18.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:18.406 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-146028888069 2026-02-25T22:29:18.406 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:18.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:29:18.408 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-146028888069 2026-02-25T22:29:18.408 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:18.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=146028888069 2026-02-25T22:29:18.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 146028888069' 2026-02-25T22:29:18.410 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 146028888069 2026-02-25T22:29:18.410 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:29:18.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 146028888069 -lt 146028888069 2026-02-25T22:29:18.624 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:18.625 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-180388626436 2026-02-25T22:29:18.625 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:18.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:29:18.626 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-180388626436 2026-02-25T22:29:18.626 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:18.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=180388626436 2026-02-25T22:29:18.628 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 180388626436 2026-02-25T22:29:18.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 180388626436' 2026-02-25T22:29:18.629 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:29:18.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 180388626437 -lt 180388626436 2026-02-25T22:29:18.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:18.843 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-214748364803 2026-02-25T22:29:18.843 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:18.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:29:18.845 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-214748364803 2026-02-25T22:29:18.845 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:18.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=214748364803 2026-02-25T22:29:18.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 214748364803' 2026-02-25T22:29:18.846 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 214748364803 2026-02-25T22:29:18.847 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:29:19.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 214748364804 -lt 214748364803 2026-02-25T22:29:19.074 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:29:19.074 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:19.075 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:19.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:29:19.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:29:19.359 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:29:19.359 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:29:19.359 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:29:19.359 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:29:19.359 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:29:19.359 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:29:19.568 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:29:19.568 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:29:19.569 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:19.569 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:19.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:29:19.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:29:19.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:19.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:29:19.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:19.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:516: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2026-02-25T22:29:19.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:29:19.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:29:19.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:29:19.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:29:19.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:29:19.914 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:29:20.117 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:29:20.117 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:29:20.117 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:29:20.117 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:29:20.118 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:29:20.118 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2026-02-25T22:29:20.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2026-02-25T22:29:20.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:29:20.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T22:29:20.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:29:20.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T22:29:20.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:29:20.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T22:29:20.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:29:20.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T22:29:20.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:29:20.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:29:20.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:29:20.120 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T22:29:20.120 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:29:20.120 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:29:20.120 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:20.121 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:20.121 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:20.121 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:29:20.121 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:29:20.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:29:20.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:29:20.183 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:29:20.183 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T22:29:20.183 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:29:20.183 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:29:20.184 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:20.184 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:20.184 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:20.184 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:29:20.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:29:20.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2026-02-25T22:29:20.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2026-02-25T22:29:20.501 INFO:tasks.workunit.client.0.trial125.stderr:osd.0 is already out. 2026-02-25T22:29:20.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2026-02-25T22:29:20.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:29:20.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:29:20.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:29:20.512 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:29:20.512 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:29:20.512 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:29:20.513 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:29:20.513 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:29:20.513 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:29:20.622 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:29:20.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:29:20.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:29:20.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:29:20.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:29:20.623 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:29:20.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:29:20.835 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:29:20.835 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:29:20.835 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:29:20.835 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:29:20.835 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:29:20.835 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:29:20.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:29:20.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:20.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:29:20.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803787 2026-02-25T22:29:20.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803787 2026-02-25T22:29:20.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787' 2026-02-25T22:29:20.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:20.913 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:29:20.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607562 2026-02-25T22:29:20.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607562 2026-02-25T22:29:20.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-51539607562' 2026-02-25T22:29:20.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:20.994 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:29:21.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411338 2026-02-25T22:29:21.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411338 2026-02-25T22:29:21.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-51539607562 2-77309411338' 2026-02-25T22:29:21.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:21.073 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:29:21.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149705 2026-02-25T22:29:21.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149705 2026-02-25T22:29:21.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-51539607562 2-77309411338 3-111669149705' 2026-02-25T22:29:21.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:21.151 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:29:21.229 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888072 2026-02-25T22:29:21.229 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888072 2026-02-25T22:29:21.229 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-51539607562 2-77309411338 3-111669149705 4-146028888072' 2026-02-25T22:29:21.229 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:21.229 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:29:21.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626439 2026-02-25T22:29:21.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626439 2026-02-25T22:29:21.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-51539607562 2-77309411338 3-111669149705 4-146028888072 5-180388626439' 2026-02-25T22:29:21.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:21.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:29:21.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364806 2026-02-25T22:29:21.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364806 2026-02-25T22:29:21.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-51539607562 2-77309411338 3-111669149705 4-146028888072 5-180388626439 6-214748364806' 2026-02-25T22:29:21.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:21.389 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803787 2026-02-25T22:29:21.389 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:21.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:29:21.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803787 2026-02-25T22:29:21.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:21.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803787 2026-02-25T22:29:21.393 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803787 2026-02-25T22:29:21.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803787' 2026-02-25T22:29:21.393 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:21.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 0 -lt 25769803787 2026-02-25T22:29:21.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:29:22.609 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:29:22.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:22.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803787 -lt 25769803787 2026-02-25T22:29:22.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:22.827 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607562 2026-02-25T22:29:22.827 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:22.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:29:22.829 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607562 2026-02-25T22:29:22.829 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:22.830 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607562 2026-02-25T22:29:22.830 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 51539607562 2026-02-25T22:29:22.831 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607562' 2026-02-25T22:29:22.831 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:29:23.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607562 -lt 51539607562 2026-02-25T22:29:23.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:23.051 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411338 2026-02-25T22:29:23.051 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:23.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:29:23.053 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411338 2026-02-25T22:29:23.053 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:23.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411338 2026-02-25T22:29:23.054 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 77309411338 2026-02-25T22:29:23.055 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411338' 2026-02-25T22:29:23.055 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:29:23.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411338 -lt 77309411338 2026-02-25T22:29:23.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:23.273 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149705 2026-02-25T22:29:23.274 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:23.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:29:23.275 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149705 2026-02-25T22:29:23.275 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:23.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149705 2026-02-25T22:29:23.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149705' 2026-02-25T22:29:23.277 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149705 2026-02-25T22:29:23.278 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:29:23.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149705 -lt 111669149705 2026-02-25T22:29:23.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:23.498 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-146028888072 2026-02-25T22:29:23.499 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:23.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:29:23.501 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-146028888072 2026-02-25T22:29:23.501 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:23.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=146028888072 2026-02-25T22:29:23.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 146028888072' 2026-02-25T22:29:23.502 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 146028888072 2026-02-25T22:29:23.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:29:23.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 146028888072 -lt 146028888072 2026-02-25T22:29:23.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:23.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-180388626439 2026-02-25T22:29:23.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:23.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:29:23.724 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-180388626439 2026-02-25T22:29:23.725 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:23.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=180388626439 2026-02-25T22:29:23.726 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 180388626439 2026-02-25T22:29:23.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 180388626439' 2026-02-25T22:29:23.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:29:23.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 180388626439 -lt 180388626439 2026-02-25T22:29:23.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:23.942 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-214748364806 2026-02-25T22:29:23.942 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:23.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:29:23.944 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-214748364806 2026-02-25T22:29:23.944 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:23.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=214748364806 2026-02-25T22:29:23.946 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 214748364806 2026-02-25T22:29:23.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 214748364806' 2026-02-25T22:29:23.946 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:29:24.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 214748364807 -lt 214748364806 2026-02-25T22:29:24.168 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:29:24.169 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:24.169 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:24.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:29:24.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:29:24.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:29:24.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:29:24.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:29:24.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:29:24.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:29:24.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:29:24.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:29:24.667 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:29:24.667 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:24.667 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:24.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:29:24.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:29:24.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:29:24.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2026-02-25T22:29:24.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:29:24.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:29:24.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2026-02-25T22:29:24.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:29:24.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:29:24.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2026-02-25T22:29:24.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:29:24.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:29:24.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2026-02-25T22:29:24.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:29:24.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:29:24.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2026-02-25T22:29:24.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:29:24.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:29:24.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2026-02-25T22:29:24.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:29:24.992 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:29:24.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:524: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2026-02-25T22:29:24.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:29:24.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:29:24.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2026-02-25T22:29:24.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:29:24.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:29:24.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2026-02-25T22:29:25.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:29:25.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:29:25.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:526: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:29:25.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:29:25.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:29:25.281 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:29:25.291 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:29:25.585 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:29:25.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:29:25.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:29:25.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:25.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:29:25.598 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:25.598 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:25.598 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:25.598 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:25.598 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:25.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:25.912 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:25.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:25.914 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:25.915 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:25.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:25.916 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:25.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:25.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:25.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:25.918 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:25.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:25.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:25.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:29:25.941 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:25.941 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:25.941 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:25.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:29:25.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:25.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:25.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:29:25.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:29:25.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:29:25.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:29:25.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:29:25.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:25.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:29:25.944 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:25.944 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:25.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:25.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:25.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:25.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:25.946 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:25.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:25.947 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:25.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:25.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:25.949 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:25.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:25.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:25.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:25.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:25.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:25.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:25.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:29:25.954 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:25.954 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:25.954 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:25.954 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:29:25.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:25.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:25.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:29:25.957 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:29:25.957 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:25.957 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:25.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:29:25.959 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:29:25.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:29:25.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:29:25.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:29:25.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:29:25.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:29:25.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:29:25.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:29:25.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:29:25.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:29:25.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:29:25.983 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:29:25.983 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:25.983 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:25.984 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:25.984 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:25.984 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:25.984 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:25.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:29:26.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:29:26.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:29:26.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:26.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:26.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:29:26.016 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:29:26.022 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:26.023 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:26.023 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:26.023 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:26.024 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:26.024 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:26.024 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:29:26.024 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:26.024 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:29:26.073 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:29:26.073 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:26.074 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:26.074 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:29:26.074 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:29:26.074 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:26.074 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:26.074 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:26.074 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:26.075 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:26.075 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:26.075 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:29:26.075 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:26.075 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:29:26.127 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:29:26.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:29:26.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:29:26.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:29:26.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:29:26.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:29:26.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:29:26.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:29:26.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:26.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:26.239 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:26.239 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:26.239 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:26.240 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:26.240 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:26.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:26.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:29:26.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:29:26.374 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:29:26.381 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:29:27.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:29:27.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:27.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:27.384 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:27.385 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:27.385 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:27.385 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:29:27.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:29:27.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:29:27.433 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:29:27.439 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:29:27.440 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:29:26.003+0000 7f50823c1ec0 0 load: jerasure load: lrc 2026-02-25T22:29:27.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2026-02-25T22:29:27.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2026-02-25T22:29:27.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: local objname=myobject 2026-02-25T22:29:27.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: local lastobj=100 2026-02-25T22:29:27.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:633: TEST_ec_recovery_unfound: local testobj=obj75 2026-02-25T22:29:27.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:635: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:29:27.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T22:29:27.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:29:27.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:638: TEST_ec_recovery_unfound: setup_osds 5 2026-02-25T22:29:27.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2026-02-25T22:29:27.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:29:27.442 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2026-02-25T22:29:27.443 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2026-02-25T22:29:27.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:29:27.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:29:27.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:29:27.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:27.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:29:27.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:29:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:29:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:29:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:29:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:27.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:29:27.447 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:27.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:27.449 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:27.449 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:27.449 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:27.449 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:27.449 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:29:27.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:27.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b40e27b5-9b58-416e-83d4-cc113fc2fd84 2026-02-25T22:29:27.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b40e27b5-9b58-416e-83d4-cc113fc2fd84' 2026-02-25T22:29:27.450 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 b40e27b5-9b58-416e-83d4-cc113fc2fd84 2026-02-25T22:29:27.450 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:27.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDHd59pbkpaGxAAYaaRLmxrxk+8vAKhxCwSmQ== 2026-02-25T22:29:27.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDHd59pbkpaGxAAYaaRLmxrxk+8vAKhxCwSmQ=="}' 2026-02-25T22:29:27.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b40e27b5-9b58-416e-83d4-cc113fc2fd84 -i td/test-erasure-eio/0/new.json 2026-02-25T22:29:27.574 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:29:27.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:29:27.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDHd59pbkpaGxAAYaaRLmxrxk+8vAKhxCwSmQ== --osd-uuid b40e27b5-9b58-416e-83d4-cc113fc2fd84 2026-02-25T22:29:27.602 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:27.598+0000 7fd359a97a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:27.604 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:27.602+0000 7fd359a97a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:27.604 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:27.602+0000 7fd359a97a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:27.605 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:27.602+0000 7fd359a97a40 -1 bdev(0x5628eb790400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:27.605 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:27.602+0000 7fd359a97a40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:29:28.318 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:29:28.318 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:28.319 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:29:28.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:29:28.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:28.461 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:29:28.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:29:28.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:28.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:28.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:28.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:28.487 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:28.486+0000 7f4871bc1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:28.489 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:28.486+0000 7f4871bc1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:28.490 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:28.486+0000 7f4871bc1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:28.570 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:29:28.570 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:28.570 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:29:28.571 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:28.571 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:29:28.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:28.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:28.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:28.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:28.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:28.786 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:28.873 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:28.870+0000 7f4871bc1a40 -1 Falling back to public interface 2026-02-25T22:29:29.457 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:29.454+0000 7f4871bc1a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:29:29.787 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:29.788 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:29.788 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:29:29.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:29.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:29.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:30.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:31.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:31.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:31.009 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:29:31.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:31.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:31.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:31.227 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1869267544,v1:127.0.0.1:6803/1869267544] [v2:127.0.0.1:6804/1869267544,v1:127.0.0.1:6805/1869267544] exists,up b40e27b5-9b58-416e-83d4-cc113fc2fd84 2026-02-25T22:29:31.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:31.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:31.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:31.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:29:31.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:29:31.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:29:31.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:31.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:29:31.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:31.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:29:31.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:29:31.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:31.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:31.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:31.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:29:31.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:29:31.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:31.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:31.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:29:31.233 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:31.234 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:31.234 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:31.234 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:31.234 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:31.234 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:31.234 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:31.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:31.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:31.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:31.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:31.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:29:31.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:29:31.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:31.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:31.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:31.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:31.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:31.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:31.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:29:31.235 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:31.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3468abff-143c-4692-b18c-5033361d3756 2026-02-25T22:29:31.236 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 3468abff-143c-4692-b18c-5033361d3756 2026-02-25T22:29:31.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3468abff-143c-4692-b18c-5033361d3756' 2026-02-25T22:29:31.237 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:31.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDLd59pQjaPDhAAPskR7rzhjG2H++dVVzq3bw== 2026-02-25T22:29:31.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDLd59pQjaPDhAAPskR7rzhjG2H++dVVzq3bw=="}' 2026-02-25T22:29:31.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3468abff-143c-4692-b18c-5033361d3756 -i td/test-erasure-eio/1/new.json 2026-02-25T22:29:31.467 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:29:31.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:29:31.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLd59pQjaPDhAAPskR7rzhjG2H++dVVzq3bw== --osd-uuid 3468abff-143c-4692-b18c-5033361d3756 2026-02-25T22:29:31.496 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:31.494+0000 7f30883d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:31.498 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:31.494+0000 7f30883d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:31.499 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:31.494+0000 7f30883d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:31.499 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:31.494+0000 7f30883d7a40 -1 bdev(0x55e0ac434400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:31.499 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:31.498+0000 7f30883d7a40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:29:32.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:29:32.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:32.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:29:32.156 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:29:32.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:32.446 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:29:32.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:29:32.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:32.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:32.447 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:32.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:32.474 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:32.470+0000 7fceaf285a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:32.476 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:32.474+0000 7fceaf285a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:32.477 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:32.474+0000 7fceaf285a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:32.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:29:32.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:32.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:29:32.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:32.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:32.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:32.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:32.656 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:29:32.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:32.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:29:32.813 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:32.810+0000 7fceaf285a40 -1 Falling back to public interface 2026-02-25T22:29:32.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:33.376 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:33.374+0000 7fceaf285a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:29:33.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:33.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:33.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:33.869 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:29:33.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:33.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:29:34.082 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2767051274,v1:127.0.0.1:6811/2767051274] [v2:127.0.0.1:6812/2767051274,v1:127.0.0.1:6813/2767051274] exists,up 3468abff-143c-4692-b18c-5033361d3756 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:29:34.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:34.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:34.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:34.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:29:34.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:29:34.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:34.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:34.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:34.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:34.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:34.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:34.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:29:34.088 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:34.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=317cff13-d82f-4d7a-9bd5-7976dd69d790 2026-02-25T22:29:34.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 317cff13-d82f-4d7a-9bd5-7976dd69d790' 2026-02-25T22:29:34.089 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 317cff13-d82f-4d7a-9bd5-7976dd69d790 2026-02-25T22:29:34.089 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:34.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDOd59pe1j2BRAAyzoMYtwBfmg/895jE8xt4A== 2026-02-25T22:29:34.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDOd59pe1j2BRAAyzoMYtwBfmg/895jE8xt4A=="}' 2026-02-25T22:29:34.102 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 317cff13-d82f-4d7a-9bd5-7976dd69d790 -i td/test-erasure-eio/2/new.json 2026-02-25T22:29:34.325 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:29:34.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:29:34.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOd59pe1j2BRAAyzoMYtwBfmg/895jE8xt4A== --osd-uuid 317cff13-d82f-4d7a-9bd5-7976dd69d790 2026-02-25T22:29:34.355 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:34.350+0000 7f5870bffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:34.356 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:34.354+0000 7f5870bffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:34.357 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:34.354+0000 7f5870bffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:34.358 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:34.354+0000 7f5870bffa40 -1 bdev(0x55d5c0096400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:34.358 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:34.354+0000 7f5870bffa40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:29:35.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:29:35.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:35.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:29:35.016 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:29:35.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:35.317 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:29:35.318 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:29:35.318 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:35.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:35.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:35.319 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:35.345 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:35.342+0000 7f00d5d7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:35.347 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:35.346+0000 7f00d5d7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:35.348 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:35.346+0000 7f00d5d7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:35.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:29:35.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:35.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:29:35.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:35.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:35.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:35.529 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:29:35.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:35.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:35.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:29:35.729 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:35.726+0000 7f00d5d7aa40 -1 Falling back to public interface 2026-02-25T22:29:35.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:36.317 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:36.314+0000 7f00d5d7aa40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:29:36.748 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:36.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:36.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:36.749 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:29:36.750 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:36.750 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:29:36.966 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1659240550,v1:127.0.0.1:6819/1659240550] [v2:127.0.0.1:6820/1659240550,v1:127.0.0.1:6821/1659240550] exists,up 317cff13-d82f-4d7a-9bd5-7976dd69d790 2026-02-25T22:29:36.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:36.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:36.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:36.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:29:36.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:29:36.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:29:36.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:36.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:29:36.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:36.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:29:36.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:29:36.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:36.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:36.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:36.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:29:36.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:29:36.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:36.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:36.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:29:36.973 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:36.973 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:36.973 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:36.973 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:36.973 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:36.973 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:36.974 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:36.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:36.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:36.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:36.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:36.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:29:36.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:29:36.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:36.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:36.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:36.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:36.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:36.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:36.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:29:36.975 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:36.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6e1adfe9-6054-4c6d-9c82-6e2a19e98dca 2026-02-25T22:29:36.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 6e1adfe9-6054-4c6d-9c82-6e2a19e98dca' 2026-02-25T22:29:36.976 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 6e1adfe9-6054-4c6d-9c82-6e2a19e98dca 2026-02-25T22:29:36.976 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:36.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDQd59paEaOOhAAi+VSUF2W9uYoQGXwrTSoCQ== 2026-02-25T22:29:36.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDQd59paEaOOhAAi+VSUF2W9uYoQGXwrTSoCQ=="}' 2026-02-25T22:29:36.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6e1adfe9-6054-4c6d-9c82-6e2a19e98dca -i td/test-erasure-eio/3/new.json 2026-02-25T22:29:37.202 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:29:37.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:29:37.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQd59paEaOOhAAi+VSUF2W9uYoQGXwrTSoCQ== --osd-uuid 6e1adfe9-6054-4c6d-9c82-6e2a19e98dca 2026-02-25T22:29:37.241 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:37.238+0000 7f9e5a2eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:37.242 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:37.238+0000 7f9e5a2eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:37.243 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:37.242+0000 7f9e5a2eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:37.244 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:37.242+0000 7f9e5a2eaa40 -1 bdev(0x558c6d04c400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:37.244 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:37.242+0000 7f9e5a2eaa40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:29:37.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:29:37.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:37.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:29:37.899 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:29:37.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:38.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:29:38.185 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:29:38.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:38.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:38.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:38.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:38.212 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:38.210+0000 7f769b2fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:38.214 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:38.210+0000 7f769b2fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:38.214 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:38.210+0000 7f769b2fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:38.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:29:38.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:38.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:29:38.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:38.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:38.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:38.399 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:29:38.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:38.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:38.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:29:38.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:38.657 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:38.654+0000 7f769b2fda40 -1 Falling back to public interface 2026-02-25T22:29:39.265 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:39.262+0000 7f769b2fda40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:29:39.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:39.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:39.615 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:29:39.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:39.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:39.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:29:39.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:40.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:40.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:40.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:40.845 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:29:40.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:40.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:29:41.054 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 26 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3610427428,v1:127.0.0.1:6827/3610427428] [v2:127.0.0.1:6828/3610427428,v1:127.0.0.1:6829/3610427428] exists,up 6e1adfe9-6054-4c6d-9c82-6e2a19e98dca 2026-02-25T22:29:41.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:41.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:41.060 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:41.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:29:41.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T22:29:41.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:29:41.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:41.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T22:29:41.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:41.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T22:29:41.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T22:29:41.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:41.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:41.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:41.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T22:29:41.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T22:29:41.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:41.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:41.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:29:41.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:41.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:41.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:41.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:41.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:41.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:41.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:41.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:41.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:41.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:41.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:41.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:29:41.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:29:41.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:41.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:41.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:41.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:41.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:41.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:41.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T22:29:41.065 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:41.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d09cc071-71d2-453c-a678-22f5a3c78d71 2026-02-25T22:29:41.065 INFO:tasks.workunit.client.0.trial125.stdout:add osd4 d09cc071-71d2-453c-a678-22f5a3c78d71 2026-02-25T22:29:41.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 d09cc071-71d2-453c-a678-22f5a3c78d71' 2026-02-25T22:29:41.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:41.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDVd59pFNtNBBAANCc4QG7mxs3jz9fUXJSUPA== 2026-02-25T22:29:41.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDVd59pFNtNBBAANCc4QG7mxs3jz9fUXJSUPA=="}' 2026-02-25T22:29:41.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d09cc071-71d2-453c-a678-22f5a3c78d71 -i td/test-erasure-eio/4/new.json 2026-02-25T22:29:41.300 INFO:tasks.workunit.client.0.trial125.stdout:4 2026-02-25T22:29:41.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T22:29:41.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVd59pFNtNBBAANCc4QG7mxs3jz9fUXJSUPA== --osd-uuid d09cc071-71d2-453c-a678-22f5a3c78d71 2026-02-25T22:29:41.337 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:41.334+0000 7f23f3c23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:41.339 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:41.334+0000 7f23f3c23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:41.340 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:41.338+0000 7f23f3c23a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:41.341 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:41.338+0000 7f23f3c23a40 -1 bdev(0x55b8fbcbe400 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:41.341 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:41.338+0000 7f23f3c23a40 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T22:29:42.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T22:29:42.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:42.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T22:29:42.043 INFO:tasks.workunit.client.0.trial125.stdout:adding osd4 key to auth repository 2026-02-25T22:29:42.044 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:42.335 INFO:tasks.workunit.client.0.trial125.stdout:start osd.4 2026-02-25T22:29:42.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T22:29:42.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:42.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:42.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:42.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:42.364 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:42.362+0000 7fef818c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:42.366 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:42.362+0000 7fef818c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:42.367 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:42.362+0000 7fef818c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:42.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T22:29:42.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:42.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T22:29:42.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:42.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:42.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:42.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:42.552 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:29:42.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:42.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:29:42.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:42.805 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:42.802+0000 7fef818c1a40 -1 Falling back to public interface 2026-02-25T22:29:43.395 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:43.390+0000 7fef818c1a40 -1 osd.4 0 log_to_monitors true 2026-02-25T22:29:43.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:43.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:43.769 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:29:43.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:43.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:43.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:29:43.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:44.983 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:44.983 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:44.984 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:29:44.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:44.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:44.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:29:45.202 INFO:tasks.workunit.client.0.trial125.stdout:osd.4 up in weight 1 up_from 34 up_thru 35 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2702135693,v1:127.0.0.1:6835/2702135693] [v2:127.0.0.1:6836/2702135693,v1:127.0.0.1:6837/2702135693] exists,up d09cc071-71d2-453c-a678-22f5a3c78d71 2026-02-25T22:29:45.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:45.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:45.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:45.204 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:29:45.204 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:29:45.204 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:29:45.204 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:45.205 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:45.205 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:45.205 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:29:45.205 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:29:45.205 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:29:45.259 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:29:45.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:29:45.267 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:29:28.874+0000 7f4871bc1a40 0 load: jerasure load: lrc 2026-02-25T22:29:45.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:29:45.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:641: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2026-02-25T22:29:45.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T22:29:45.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:29:45.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:29:45.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T22:29:45.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:29:45.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T22:29:45.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:29:45.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T22:29:45.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:29:45.540 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:29:45.897 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:29:45.907 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:29:46.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:29:46.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:29:46.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:29:46.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:29:46.909 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:29:46.910 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:29:46.910 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:29:46.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:29:46.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:29:46.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:29:47.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:29:47.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:29:47.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:29:47.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:29:47.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:29:47.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:29:47.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:29:47.227 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:29:47.228 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:29:47.228 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:29:47.228 INFO:tasks.workunit.client.0.trial125.stderr:4' 2026-02-25T22:29:47.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:29:47.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:47.228 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:29:47.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803783 2026-02-25T22:29:47.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803783 2026-02-25T22:29:47.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783' 2026-02-25T22:29:47.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:47.311 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:29:47.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607557 2026-02-25T22:29:47.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607557 2026-02-25T22:29:47.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607557' 2026-02-25T22:29:47.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:47.389 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:29:47.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411332 2026-02-25T22:29:47.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411332 2026-02-25T22:29:47.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607557 2-77309411332' 2026-02-25T22:29:47.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:47.466 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:29:47.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149700 2026-02-25T22:29:47.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149700 2026-02-25T22:29:47.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607557 2-77309411332 3-111669149700' 2026-02-25T22:29:47.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:47.545 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:29:47.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888067 2026-02-25T22:29:47.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888067 2026-02-25T22:29:47.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607557 2-77309411332 3-111669149700 4-146028888067' 2026-02-25T22:29:47.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:47.627 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803783 2026-02-25T22:29:47.627 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:47.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:29:47.629 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803783 2026-02-25T22:29:47.629 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:47.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803783 2026-02-25T22:29:47.630 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803783 2026-02-25T22:29:47.631 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803783' 2026-02-25T22:29:47.631 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:47.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803783 2026-02-25T22:29:47.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:29:48.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:29:48.846 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:49.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803783 -lt 25769803783 2026-02-25T22:29:49.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:49.052 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607557 2026-02-25T22:29:49.053 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:49.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:29:49.055 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607557 2026-02-25T22:29:49.055 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:49.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607557 2026-02-25T22:29:49.056 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 51539607557 2026-02-25T22:29:49.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607557' 2026-02-25T22:29:49.057 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:29:49.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607557 -lt 51539607557 2026-02-25T22:29:49.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:49.273 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411332 2026-02-25T22:29:49.273 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:49.274 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:29:49.274 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411332 2026-02-25T22:29:49.275 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:49.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411332 2026-02-25T22:29:49.276 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 77309411332 2026-02-25T22:29:49.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411332' 2026-02-25T22:29:49.277 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:29:49.491 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411333 -lt 77309411332 2026-02-25T22:29:49.491 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:49.492 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149700 2026-02-25T22:29:49.492 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:49.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:29:49.494 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149700 2026-02-25T22:29:49.495 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:49.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149700 2026-02-25T22:29:49.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149700' 2026-02-25T22:29:49.496 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149700 2026-02-25T22:29:49.497 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:29:49.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149700 -lt 111669149700 2026-02-25T22:29:49.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:49.710 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-146028888067 2026-02-25T22:29:49.710 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:49.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:29:49.712 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-146028888067 2026-02-25T22:29:49.712 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:49.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=146028888067 2026-02-25T22:29:49.714 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 146028888067 2026-02-25T22:29:49.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 146028888067' 2026-02-25T22:29:49.714 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:29:49.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 146028888067 -lt 146028888067 2026-02-25T22:29:49.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:29:49.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:49.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:50.206 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:29:50.206 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:29:50.206 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:29:50.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:29:50.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:29:50.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:29:50.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:29:50.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:29:50.413 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:29:50.414 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:29:50.414 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:50.415 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:50.689 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:29:50.690 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:29:50.690 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:29:50.690 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:644: TEST_ec_recovery_unfound: ceph pg dump pgs 2026-02-25T22:29:50.889 INFO:tasks.workunit.client.0.trial125.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T22:29:50.889 INFO:tasks.workunit.client.0.trial125.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:46.320334+0000 0'0 42:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-25T22:29:45.844496+0000 0'0 2026-02-25T22:29:45.844496+0000 0 0 periodic scrub scheduled @ 2026-02-27T08:46:05.221967+0000 0 0 2026-02-25T22:29:50.889 INFO:tasks.workunit.client.0.trial125.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:44.366877+0000 0'0 42:22 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T04:21:56.138642+0000 0 0 2026-02-25T22:29:50.889 INFO:tasks.workunit.client.0.trial125.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:44.366782+0000 0'0 42:99 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T05:29:53.770058+0000 0 0 2026-02-25T22:29:50.889 INFO:tasks.workunit.client.0.trial125.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:44.567623+0000 0'0 42:22 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T01:16:44.779151+0000 0 0 2026-02-25T22:29:50.889 INFO:tasks.workunit.client.0.trial125.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:37.249096+0000 0'0 42:65 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T01:55:55.695711+0000 0 0 2026-02-25T22:29:50.889 INFO:tasks.workunit.client.0.trial125.stdout: 2026-02-25T22:29:50.890 INFO:tasks.workunit.client.0.trial125.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T22:29:50.890 INFO:tasks.workunit.client.0.trial125.stderr:dumped pgs 2026-02-25T22:29:50.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-25T22:29:50.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:29:50.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:29:50.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-25T22:29:50.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:50.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:29:50.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:50.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:29:50.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:50.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:29:50.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:29:50.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:29:50.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:50.919 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2026-02-25T22:29:50.919 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:29:50.919 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T22:29:50.920 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:29:50.920 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:29:51.136 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:29:51.136 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:29:51.136 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:29:51.136 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:29:51.136 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:29:51.136 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2 2026-02-25T22:29:51.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: initial_osds=('3' '1' '4' '0' '2') 2026-02-25T22:29:51.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: local -a initial_osds 2026-02-25T22:29:51.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:649: TEST_ec_recovery_unfound: local last_osd=2 2026-02-25T22:29:51.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T22:29:51.138 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:51.138 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:51.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:51.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:51.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:51.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:51.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: ceph osd down 2 2026-02-25T22:29:51.656 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 is already down. 2026-02-25T22:29:51.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: ceph osd out 2 2026-02-25T22:29:51.907 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 is already out. 2026-02-25T22:29:51.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: ceph pg dump pgs 2026-02-25T22:29:52.125 INFO:tasks.workunit.client.0.trial125.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T22:29:52.125 INFO:tasks.workunit.client.0.trial125.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:46.320334+0000 0'0 42:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-25T22:29:45.844496+0000 0'0 2026-02-25T22:29:45.844496+0000 0 0 periodic scrub scheduled @ 2026-02-27T08:46:05.221967+0000 0 0 2026-02-25T22:29:52.126 INFO:tasks.workunit.client.0.trial125.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:44.366877+0000 0'0 42:22 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T04:21:56.138642+0000 0 0 2026-02-25T22:29:52.126 INFO:tasks.workunit.client.0.trial125.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:44.366782+0000 0'0 42:99 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T05:29:53.770058+0000 0 0 2026-02-25T22:29:52.126 INFO:tasks.workunit.client.0.trial125.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:44.567623+0000 0'0 42:22 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T01:16:44.779151+0000 0 0 2026-02-25T22:29:52.126 INFO:tasks.workunit.client.0.trial125.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:37.249096+0000 0'0 42:65 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T01:55:55.695711+0000 0 0 2026-02-25T22:29:52.126 INFO:tasks.workunit.client.0.trial125.stdout: 2026-02-25T22:29:52.126 INFO:tasks.workunit.client.0.trial125.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T22:29:52.126 INFO:tasks.workunit.client.0.trial125.stderr:dumped pgs 2026-02-25T22:29:52.136 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:656: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2026-02-25T22:29:52.137 INFO:tasks.workunit.client.0.trial125.stderr:4+0 records in 2026-02-25T22:29:52.137 INFO:tasks.workunit.client.0.trial125.stderr:4+0 records out 2026-02-25T22:29:52.137 INFO:tasks.workunit.client.0.trial125.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000110465 s, 37.1 MB/s 2026-02-25T22:29:52.138 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: seq 1 100 2026-02-25T22:29:52.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.476 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.476 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.517 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.588 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.631 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.631 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.734 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:52.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:52.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.204 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.204 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.322 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.782 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.782 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.825 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.825 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.924 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.924 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.950 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:53.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:53.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.080 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.080 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.105 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.105 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.323 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:29:54.531 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2026-02-25T22:29:54.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2026-02-25T22:29:54.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:29:54.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:29:54.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:29:54.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:29:54.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:29:54.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:29:54.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj75 2026-02-25T22:29:54.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:29:54.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:29:54.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:29:54.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T22:29:54.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:29:54.558 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj75 2026-02-25T22:29:54.558 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:29:54.558 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj75 2026-02-25T22:29:54.559 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj75 2026-02-25T22:29:54.559 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:29:54.768 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:29:54.768 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:29:54.768 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:29:54.768 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:29:54.768 INFO:tasks.workunit.client.0.trial125.stderr:2147483647' 2026-02-25T22:29:54.768 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2026-02-25T22:29:54.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-25T22:29:54.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:29:54.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T22:29:54.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:29:54.770 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T22:29:54.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:29:54.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T22:29:54.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:29:54.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T22:29:54.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:29:54.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:29:54.771 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:29:54.772 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T22:29:54.772 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:29:54.772 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:29:54.772 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:54.772 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:54.772 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:54.772 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:29:54.772 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:29:54.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:29:54.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:29:54.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:29:54.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T22:29:54.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:29:54.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:29:54.836 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:54.836 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:54.836 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:54.837 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:29:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:29:54.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2026-02-25T22:29:54.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:663: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2026-02-25T22:29:54.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:29:54.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:29:54.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:29:54.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:29:54.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:29:54.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:29:54.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj75 2026-02-25T22:29:54.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:29:54.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:29:54.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:29:54.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T22:29:54.899 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:29:54.899 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj75 2026-02-25T22:29:54.899 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:29:54.899 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj75 2026-02-25T22:29:54.899 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj75 2026-02-25T22:29:54.899 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:29:55.108 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:29:55.108 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:29:55.108 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:29:55.108 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:29:55.108 INFO:tasks.workunit.client.0.trial125.stderr:2147483647' 2026-02-25T22:29:55.108 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2026-02-25T22:29:55.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-25T22:29:55.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:29:55.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=1 2026-02-25T22:29:55.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:29:55.110 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/1/type 2026-02-25T22:29:55.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:29:55.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-25T22:29:55.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:29:55.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-25T22:29:55.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:29:55.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:29:55.111 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:29:55.111 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-25T22:29:55.112 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:29:55.112 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:29:55.112 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:55.112 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:55.112 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:55.112 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.1.asok 2026-02-25T22:29:55.112 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:29:55.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:29:55.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:29:55.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:29:55.175 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.1 2026-02-25T22:29:55.175 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:29:55.176 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:29:55.176 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:55.176 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:55.176 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:55.176 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.1.asok 2026-02-25T22:29:55.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:29:55.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2026-02-25T22:29:55.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:665: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2026-02-25T22:29:55.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:29:55.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:29:55.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T22:29:55.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:29:55.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:55.239 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:55.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:55.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:29:55.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:29:55.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:29:55.241 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:29:55.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T22:29:55.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:55.242 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T22:29:55.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T22:29:55.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:29:55.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:29:55.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:29:55.261 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:55.258+0000 7f1aad3e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:55.262 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:55.258+0000 7f1aad3e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:55.263 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:55.262+0000 7f1aad3e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:55.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T22:29:55.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:55.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:29:55.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:55.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:55.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:55.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:55.458 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:29:55.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:55.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:29:55.621 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:55.618+0000 7f1aad3e1a40 -1 Falling back to public interface 2026-02-25T22:29:55.669 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:56.275 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:29:56.270+0000 7f1aad3e1a40 -1 osd.2 42 log_to_monitors true 2026-02-25T22:29:56.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:56.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:56.671 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:56.671 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:29:56.671 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:56.671 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:29:56.880 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up out weight 0 up_from 48 up_thru 20 down_at 43 last_clean_interval [18,42) [v2:127.0.0.1:6818/2058387609,v1:127.0.0.1:6819/2058387609] [v2:127.0.0.1:6820/2058387609,v1:127.0.0.1:6821/2058387609] exists,up 317cff13-d82f-4d7a-9bd5-7976dd69d790 2026-02-25T22:29:56.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:56.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:56.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:56.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph osd in 2 2026-02-25T22:29:57.127 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 is already in. 2026-02-25T22:29:57.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:668: TEST_ec_recovery_unfound: sleep 15 2026-02-25T22:30:12.139 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: seq 1 100 2026-02-25T22:30:12.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2026-02-25T22:30:12.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:671: TEST_ec_recovery_unfound: get_state 2.0 2026-02-25T22:30:12.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2026-02-25T22:30:12.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2026-02-25T22:30:12.142 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2026-02-25T22:30:12.142 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2026-02-25T22:30:12.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:671: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2026-02-25T22:30:12.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:672: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2026-02-25T22:30:12.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:672: TEST_ec_recovery_unfound: grep recovery_unfound 2026-02-25T22:30:12.346 INFO:tasks.workunit.client.0.trial125.stdout:active+recovery_unfound+degraded 2026-02-25T22:30:12.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2026-02-25T22:30:12.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: break 2026-02-25T22:30:12.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: ceph pg dump pgs 2026-02-25T22:30:12.556 INFO:tasks.workunit.client.0.trial125.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T22:30:12.556 INFO:tasks.workunit.client.0.trial125.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2026-02-25T22:29:57.430968+0000 46'101 50:345 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-25T22:29:45.844496+0000 0'0 2026-02-25T22:29:45.844496+0000 0 0 no scrub is scheduled 0 0 2026-02-25T22:30:12.556 INFO:tasks.workunit.client.0.trial125.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:57.320990+0000 0'0 50:55 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T07:04:34.906475+0000 0 0 2026-02-25T22:30:12.556 INFO:tasks.workunit.client.0.trial125.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:44.366782+0000 0'0 50:115 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T05:29:53.770058+0000 0 0 2026-02-25T22:30:12.556 INFO:tasks.workunit.client.0.trial125.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:44.567623+0000 0'0 50:37 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T01:16:44.779151+0000 0 0 2026-02-25T22:30:12.557 INFO:tasks.workunit.client.0.trial125.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T22:29:57.322847+0000 0'0 50:98 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-25T22:29:26.355724+0000 0'0 2026-02-25T22:29:26.355724+0000 0 0 periodic scrub scheduled @ 2026-02-27T07:19:43.700842+0000 0 0 2026-02-25T22:30:12.557 INFO:tasks.workunit.client.0.trial125.stdout: 2026-02-25T22:30:12.557 INFO:tasks.workunit.client.0.trial125.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T22:30:12.557 INFO:tasks.workunit.client.0.trial125.stderr:dumped pgs 2026-02-25T22:30:12.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:681: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-25T22:30:12.651 INFO:tasks.workunit.client.0.trial125.stdout:{ 2026-02-25T22:30:12.651 INFO:tasks.workunit.client.0.trial125.stdout: "num_missing": 1, 2026-02-25T22:30:12.651 INFO:tasks.workunit.client.0.trial125.stdout: "num_unfound": 1, 2026-02-25T22:30:12.651 INFO:tasks.workunit.client.0.trial125.stdout: "objects": [ 2026-02-25T22:30:12.651 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.651 INFO:tasks.workunit.client.0.trial125.stdout: "oid": { 2026-02-25T22:30:12.651 INFO:tasks.workunit.client.0.trial125.stdout: "oid": "obj75", 2026-02-25T22:30:12.651 INFO:tasks.workunit.client.0.trial125.stdout: "key": "", 2026-02-25T22:30:12.651 INFO:tasks.workunit.client.0.trial125.stdout: "snapid": -2, 2026-02-25T22:30:12.651 INFO:tasks.workunit.client.0.trial125.stdout: "hash": 3565005477, 2026-02-25T22:30:12.651 INFO:tasks.workunit.client.0.trial125.stdout: "max": 0, 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: "pool": 2, 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: "namespace": "" 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: "need": "46'76", 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: "have": "0'0", 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: "flags": "none", 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: "locations": [ 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: ] 2026-02-25T22:30:12.652 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.653 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.653 INFO:tasks.workunit.client.0.trial125.stdout: "state": "NotRecovering", 2026-02-25T22:30:12.653 INFO:tasks.workunit.client.0.trial125.stdout: "available_might_have_unfound": true, 2026-02-25T22:30:12.653 INFO:tasks.workunit.client.0.trial125.stdout: "might_have_unfound": [], 2026-02-25T22:30:12.653 INFO:tasks.workunit.client.0.trial125.stdout: "more": false 2026-02-25T22:30:12.653 INFO:tasks.workunit.client.0.trial125.stdout:} 2026-02-25T22:30:12.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: ceph pg 2.0 query 2026-02-25T22:30:12.740 INFO:tasks.workunit.client.0.trial125.stdout:{ 2026-02-25T22:30:12.740 INFO:tasks.workunit.client.0.trial125.stdout: "snap_trimq": "[]", 2026-02-25T22:30:12.740 INFO:tasks.workunit.client.0.trial125.stdout: "snap_trimq_len": 0, 2026-02-25T22:30:12.740 INFO:tasks.workunit.client.0.trial125.stdout: "state": "active+recovery_unfound+degraded", 2026-02-25T22:30:12.740 INFO:tasks.workunit.client.0.trial125.stdout: "epoch": 50, 2026-02-25T22:30:12.740 INFO:tasks.workunit.client.0.trial125.stdout: "up": [ 2026-02-25T22:30:12.740 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.740 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.740 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.740 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: "acting": [ 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: "acting_recovery_backfill": [ 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: "1(1)", 2026-02-25T22:30:12.741 INFO:tasks.workunit.client.0.trial125.stdout: "2(4)", 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "3(0)", 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "info": { 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s0", 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "shard": "0", 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "last_update": "46'101", 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "last_complete": "46'75", 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "log_tail": "0'0", 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "last_user_version": 101, 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "last_backfill": "MAX", 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete": [], 2026-02-25T22:30:12.742 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [], 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "history": { 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_created": 40, 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_pool_created": 40, 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 50, 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_started": 49, 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_clean": 40, 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_split": 0, 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_marked_full": 0, 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "same_up_since": 49, 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "same_interval_since": 49, 2026-02-25T22:30:12.743 INFO:tasks.workunit.client.0.trial125.stdout: "same_primary_since": 40, 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "prior_readable_until_ub": 0 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "stats": { 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "version": "46'101", 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "reported_seq": 346, 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "reported_epoch": 50, 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "state": "active+recovery_unfound+degraded", 2026-02-25T22:30:12.744 INFO:tasks.workunit.client.0.trial125.stdout: "last_fresh": "2026-02-25T22:29:57.431270+0000", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "last_change": "2026-02-25T22:29:57.430968+0000", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "last_active": "2026-02-25T22:29:57.431270+0000", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "last_peered": "2026-02-25T22:29:57.431270+0000", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean": "2026-02-25T22:29:50.916389+0000", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_active": "2026-02-25T22:29:57.323777+0000", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_peered": "2026-02-25T22:29:57.323777+0000", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "last_unstale": "2026-02-25T22:29:57.431270+0000", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "last_undegraded": "2026-02-25T22:29:57.321101+0000", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "last_fullsized": "2026-02-25T22:29:57.431270+0000", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "mapping_epoch": 49, 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "log_start": "0'0", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_start": "0'0", 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "created": 40, 2026-02-25T22:30:12.745 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "parent": "0.0", 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "parent_split_bits": 0, 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "objects_scrubbed": 0, 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "log_size": 101, 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "log_dups_size": 0, 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_size": 101, 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "stats_invalid": false, 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "dirty_stats_invalid": false, 2026-02-25T22:30:12.746 INFO:tasks.workunit.client.0.trial125.stdout: "omap_stats_invalid": false, 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_stats_invalid": false, 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "pin_stats_invalid": false, 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "manifest_stats_invalid": false, 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrimq_len": 0, 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_duration": 0, 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_schedule": "no scrub is scheduled", 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_duration": 0, 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "objects_trimmed": 0, 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrim_duration": 0, 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "stat_sum": { 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes": 413696, 2026-02-25T22:30:12.747 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects": 101, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_clones": 0, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_copies": 505, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing_on_primary": 1, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing": 0, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_degraded": 3, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_misplaced": 0, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_unfound": 1, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_dirty": 101, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_whiteouts": 0, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_read": 0, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_read_kb": 0, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_write": 101, 2026-02-25T22:30:12.748 INFO:tasks.workunit.client.0.trial125.stdout: "num_write_kb": 404, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_scrub_errors": 0, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_deep_scrub_errors": 0, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_recovered": 99, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_recovered": 405504, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_keys_recovered": 0, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_omap": 0, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush": 0, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_kb": 0, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict": 0, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_kb": 0, 2026-02-25T22:30:12.749 INFO:tasks.workunit.client.0.trial125.stdout: "num_promote": 0, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_high": 0, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_low": 0, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_some": 0, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_full": 0, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_pinned": 0, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "num_legacy_snapsets": 0, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "num_large_omap_objects": 0, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_manifest": 0, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_bytes": 0, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_keys": 0, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_repaired": 0 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.750 INFO:tasks.workunit.client.0.trial125.stdout: "up": [ 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: "acting": [ 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.751 INFO:tasks.workunit.client.0.trial125.stdout: "avail_no_missing": [ 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: "object_location_counts": [ 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: "objects": 100 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: "shards": "0(3),4(2)", 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: "objects": 1 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.752 INFO:tasks.workunit.client.0.trial125.stdout: "blocked_by": [], 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: "up_primary": 3, 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: "acting_primary": 3, 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [] 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: "empty": 0, 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: "dne": 0, 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: "incomplete": 0, 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 50, 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: "hit_set_history": { 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: "current_last_update": "0'0", 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: "history": [] 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.753 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "peer_info": [ 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "peer": "0(3)", 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s3", 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "shard": "3", 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "last_update": "46'101", 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "last_complete": "46'101", 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "log_tail": "0'0", 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "last_user_version": 101, 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "last_backfill": "MAX", 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete": [], 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [], 2026-02-25T22:30:12.754 INFO:tasks.workunit.client.0.trial125.stdout: "history": { 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_created": 40, 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_pool_created": 40, 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 50, 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_started": 49, 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_clean": 40, 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_split": 0, 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_marked_full": 0, 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "same_up_since": 49, 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "same_interval_since": 49, 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "same_primary_since": 40, 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:30:12.755 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "prior_readable_until_ub": 0 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "stats": { 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "version": "46'101", 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "reported_seq": 132, 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "reported_epoch": 46, 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "state": "active+undersized+degraded", 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "last_fresh": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "last_change": "2026-02-25T22:29:51.327114+0000", 2026-02-25T22:30:12.756 INFO:tasks.workunit.client.0.trial125.stdout: "last_active": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "last_peered": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean": "2026-02-25T22:29:50.916389+0000", 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_active": "2026-02-25T22:29:51.327114+0000", 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_peered": "2026-02-25T22:29:51.327114+0000", 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "last_unstale": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "last_undegraded": "2026-02-25T22:29:51.325671+0000", 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "last_fullsized": "2026-02-25T22:29:51.325571+0000", 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "mapping_epoch": 49, 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "log_start": "0'0", 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_start": "0'0", 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "created": 40, 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:30:12.757 INFO:tasks.workunit.client.0.trial125.stdout: "parent": "0.0", 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "parent_split_bits": 0, 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "objects_scrubbed": 0, 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "log_size": 101, 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "log_dups_size": 0, 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_size": 101, 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "stats_invalid": false, 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "dirty_stats_invalid": false, 2026-02-25T22:30:12.758 INFO:tasks.workunit.client.0.trial125.stdout: "omap_stats_invalid": false, 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_stats_invalid": false, 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "pin_stats_invalid": false, 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "manifest_stats_invalid": false, 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrimq_len": 0, 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_duration": 0, 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T07:41:57.977474+0000", 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_duration": 0, 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "objects_trimmed": 0, 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrim_duration": 0, 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "stat_sum": { 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes": 413696, 2026-02-25T22:30:12.759 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects": 101, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_clones": 0, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_copies": 505, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing": 0, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_degraded": 101, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_misplaced": 0, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_unfound": 0, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_dirty": 101, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_whiteouts": 0, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_read": 0, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_read_kb": 0, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_write": 101, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_write_kb": 404, 2026-02-25T22:30:12.760 INFO:tasks.workunit.client.0.trial125.stdout: "num_scrub_errors": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_deep_scrub_errors": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_recovered": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_recovered": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_keys_recovered": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_omap": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_kb": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_kb": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_promote": 0, 2026-02-25T22:30:12.761 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_high": 0, 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_low": 0, 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_some": 0, 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_full": 0, 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_pinned": 0, 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: "num_legacy_snapsets": 0, 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: "num_large_omap_objects": 0, 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_manifest": 0, 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_bytes": 0, 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_keys": 0, 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_repaired": 0 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: "up": [ 2026-02-25T22:30:12.762 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: "acting": [ 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.763 INFO:tasks.workunit.client.0.trial125.stdout: "avail_no_missing": [ 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: "3(0)", 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: "1(1)", 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: "object_location_counts": [ 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: "objects": 101 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: "blocked_by": [], 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: "up_primary": 3, 2026-02-25T22:30:12.764 INFO:tasks.workunit.client.0.trial125.stdout: "acting_primary": 3, 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [] 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: "empty": 0, 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: "dne": 0, 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: "incomplete": 0, 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 50, 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: "hit_set_history": { 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: "current_last_update": "0'0", 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: "history": [] 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: "peer": "1(1)", 2026-02-25T22:30:12.765 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s1", 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "shard": "1", 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "last_update": "46'101", 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "last_complete": "46'101", 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "log_tail": "0'0", 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "last_user_version": 101, 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "last_backfill": "MAX", 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete": [], 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [], 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "history": { 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_created": 40, 2026-02-25T22:30:12.766 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_pool_created": 40, 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 50, 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_started": 49, 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_clean": 40, 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_split": 0, 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_marked_full": 0, 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "same_up_since": 49, 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "same_interval_since": 49, 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "same_primary_since": 40, 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:30:12.767 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "prior_readable_until_ub": 0 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "stats": { 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "version": "46'101", 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "reported_seq": 132, 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "reported_epoch": 46, 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "state": "active+undersized+degraded", 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "last_fresh": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "last_change": "2026-02-25T22:29:51.327114+0000", 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "last_active": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "last_peered": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean": "2026-02-25T22:29:50.916389+0000", 2026-02-25T22:30:12.768 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_active": "2026-02-25T22:29:51.327114+0000", 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_peered": "2026-02-25T22:29:51.327114+0000", 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "last_unstale": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "last_undegraded": "2026-02-25T22:29:51.325671+0000", 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "last_fullsized": "2026-02-25T22:29:51.325571+0000", 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "mapping_epoch": 49, 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "log_start": "0'0", 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_start": "0'0", 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "created": 40, 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "parent": "0.0", 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "parent_split_bits": 0, 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:30:12.769 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "objects_scrubbed": 0, 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "log_size": 101, 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "log_dups_size": 0, 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_size": 101, 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "stats_invalid": false, 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "dirty_stats_invalid": false, 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "omap_stats_invalid": false, 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_stats_invalid": false, 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "pin_stats_invalid": false, 2026-02-25T22:30:12.770 INFO:tasks.workunit.client.0.trial125.stdout: "manifest_stats_invalid": false, 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrimq_len": 0, 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_duration": 0, 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T07:41:57.977474+0000", 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_duration": 0, 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "objects_trimmed": 0, 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrim_duration": 0, 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "stat_sum": { 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes": 413696, 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects": 101, 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_clones": 0, 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_copies": 505, 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T22:30:12.771 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing": 1, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_degraded": 101, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_misplaced": 0, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_unfound": 0, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_dirty": 101, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_whiteouts": 0, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_read": 0, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_read_kb": 0, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_write": 101, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_write_kb": 404, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_scrub_errors": 0, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_deep_scrub_errors": 0, 2026-02-25T22:30:12.772 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_recovered": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_recovered": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_keys_recovered": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_omap": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_kb": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_kb": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_promote": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_high": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_low": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_some": 0, 2026-02-25T22:30:12.773 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_full": 0, 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_pinned": 0, 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: "num_legacy_snapsets": 0, 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: "num_large_omap_objects": 0, 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_manifest": 0, 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_bytes": 0, 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_keys": 0, 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_repaired": 0 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: "up": [ 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.774 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: "acting": [ 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: "avail_no_missing": [ 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: "3(0)", 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: "1(1)", 2026-02-25T22:30:12.775 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: "object_location_counts": [ 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: "objects": 101 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: "blocked_by": [], 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: "up_primary": 3, 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: "acting_primary": 3, 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [] 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: "empty": 0, 2026-02-25T22:30:12.776 INFO:tasks.workunit.client.0.trial125.stdout: "dne": 0, 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "incomplete": 0, 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 50, 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "hit_set_history": { 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "current_last_update": "0'0", 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "history": [] 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "peer": "2(4)", 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s4", 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "shard": "4", 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "last_update": "46'101", 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "last_complete": "42'1", 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "log_tail": "0'0", 2026-02-25T22:30:12.777 INFO:tasks.workunit.client.0.trial125.stdout: "last_user_version": 1, 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "last_backfill": "MAX", 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete": [], 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [], 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "history": { 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_created": 40, 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_pool_created": 40, 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 50, 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_started": 49, 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_clean": 40, 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_split": 0, 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_marked_full": 0, 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "same_up_since": 49, 2026-02-25T22:30:12.778 INFO:tasks.workunit.client.0.trial125.stdout: "same_interval_since": 49, 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "same_primary_since": 40, 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "prior_readable_until_ub": 0 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "stats": { 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "version": "42'1", 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "reported_seq": 18, 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "reported_epoch": 42, 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "state": "active+clean", 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "last_fresh": "2026-02-25T22:29:50.916389+0000", 2026-02-25T22:30:12.779 INFO:tasks.workunit.client.0.trial125.stdout: "last_change": "2026-02-25T22:29:46.320334+0000", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_active": "2026-02-25T22:29:50.916389+0000", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_peered": "2026-02-25T22:29:50.916389+0000", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean": "2026-02-25T22:29:50.916389+0000", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_active": "2026-02-25T22:29:46.320044+0000", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_peered": "2026-02-25T22:29:46.320044+0000", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_unstale": "2026-02-25T22:29:50.916389+0000", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_undegraded": "2026-02-25T22:29:50.916389+0000", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_fullsized": "2026-02-25T22:29:50.916389+0000", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "mapping_epoch": 49, 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "log_start": "0'0", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_start": "0'0", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "created": 40, 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "parent": "0.0", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "parent_split_bits": 0, 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:30:12.780 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "objects_scrubbed": 0, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "log_size": 1, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "log_dups_size": 0, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_size": 1, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "stats_invalid": false, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "dirty_stats_invalid": false, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "omap_stats_invalid": false, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_stats_invalid": false, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "pin_stats_invalid": false, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "manifest_stats_invalid": false, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrimq_len": 0, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_duration": 0, 2026-02-25T22:30:12.781 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T08:46:05.221967+0000", 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_duration": 0, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "objects_trimmed": 0, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrim_duration": 0, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "stat_sum": { 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes": 4096, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects": 1, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_clones": 0, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_copies": 5, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing": 1, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_degraded": 0, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_misplaced": 0, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_unfound": 0, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_dirty": 1, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_whiteouts": 0, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_read": 0, 2026-02-25T22:30:12.782 INFO:tasks.workunit.client.0.trial125.stdout: "num_read_kb": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_write": 1, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_write_kb": 4, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_scrub_errors": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_deep_scrub_errors": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_recovered": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_recovered": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_keys_recovered": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_omap": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_kb": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_kb": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_promote": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_high": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_low": 0, 2026-02-25T22:30:12.783 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_some": 0, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_full": 0, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_pinned": 0, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: "num_legacy_snapsets": 0, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: "num_large_omap_objects": 0, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_manifest": 0, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_bytes": 0, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_keys": 0, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_repaired": 0 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: "up": [ 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: "acting": [ 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.784 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "avail_no_missing": [], 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "object_location_counts": [], 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "blocked_by": [], 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "up_primary": 3, 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "acting_primary": 3, 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [] 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "empty": 0, 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "dne": 0, 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "incomplete": 0, 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 50, 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "hit_set_history": { 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "current_last_update": "0'0", 2026-02-25T22:30:12.785 INFO:tasks.workunit.client.0.trial125.stdout: "history": [] 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "peer": "4(2)", 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s2", 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "shard": "2", 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "last_update": "46'101", 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "last_complete": "46'101", 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "log_tail": "0'0", 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "last_user_version": 101, 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "last_backfill": "MAX", 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete": [], 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [], 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "history": { 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_created": 40, 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "epoch_pool_created": 40, 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 50, 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_started": 49, 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:30:12.786 INFO:tasks.workunit.client.0.trial125.stdout: "last_interval_clean": 40, 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_split": 0, 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_marked_full": 0, 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "same_up_since": 49, 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "same_interval_since": 49, 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "same_primary_since": 40, 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "prior_readable_until_ub": 0 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "stats": { 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "version": "46'101", 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "reported_seq": 132, 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "reported_epoch": 46, 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "state": "active+undersized+degraded", 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "last_fresh": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "last_change": "2026-02-25T22:29:51.327114+0000", 2026-02-25T22:30:12.787 INFO:tasks.workunit.client.0.trial125.stdout: "last_active": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_peered": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean": "2026-02-25T22:29:50.916389+0000", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_active": "2026-02-25T22:29:51.327114+0000", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_became_peered": "2026-02-25T22:29:51.327114+0000", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_unstale": "2026-02-25T22:29:54.553455+0000", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_undegraded": "2026-02-25T22:29:51.325671+0000", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_fullsized": "2026-02-25T22:29:51.325571+0000", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "mapping_epoch": 49, 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "log_start": "0'0", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_start": "0'0", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "created": 40, 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_clean": 41, 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "parent": "0.0", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "parent_split_bits": 0, 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub": "0'0", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub": "0'0", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_deep_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "last_clean_scrub_stamp": "2026-02-25T22:29:45.844496+0000", 2026-02-25T22:30:12.788 INFO:tasks.workunit.client.0.trial125.stdout: "objects_scrubbed": 0, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "log_size": 101, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "log_dups_size": 0, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "ondisk_log_size": 101, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "stats_invalid": false, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "dirty_stats_invalid": false, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "omap_stats_invalid": false, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_stats_invalid": false, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "pin_stats_invalid": false, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "manifest_stats_invalid": false, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrimq_len": 0, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "last_scrub_duration": 0, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T07:41:57.977474+0000", 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "scrub_duration": 0, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "objects_trimmed": 0, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "snaptrim_duration": 0, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "stat_sum": { 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes": 413696, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects": 101, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_clones": 0, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "num_object_copies": 505, 2026-02-25T22:30:12.789 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_missing": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_degraded": 101, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_misplaced": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_unfound": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_dirty": 101, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_whiteouts": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_read": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_read_kb": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_write": 101, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_write_kb": 404, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_scrub_errors": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_deep_scrub_errors": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_recovered": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_recovered": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_keys_recovered": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_omap": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush": 0, 2026-02-25T22:30:12.790 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_kb": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_kb": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_promote": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_high": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_flush_mode_low": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_some": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_evict_mode_full": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_pinned": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_legacy_snapsets": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_large_omap_objects": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_manifest": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_bytes": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_omap_keys": 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "num_objects_repaired": 0 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "up": [ 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.791 INFO:tasks.workunit.client.0.trial125.stdout: "acting": [ 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: 3, 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: 1, 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: 4, 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: 0, 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: 2 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "avail_no_missing": [ 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "3(0)", 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "0(3)", 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "1(1)", 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "4(2)" 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "object_location_counts": [ 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "objects": 101 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "blocked_by": [], 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "up_primary": 3, 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "acting_primary": 3, 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: "purged_snaps": [] 2026-02-25T22:30:12.792 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "empty": 0, 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "dne": 0, 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "incomplete": 0, 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "last_epoch_started": 50, 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "hit_set_history": { 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "current_last_update": "0'0", 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "history": [] 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "recovery_state": [ 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "name": "Started/Primary/Active", 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "enter_time": "2026-02-25T22:29:57.320555+0000", 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "might_have_unfound": [ 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "osd": "0(3)", 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "status": "already probed" 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "osd": "1(1)", 2026-02-25T22:30:12.793 INFO:tasks.workunit.client.0.trial125.stdout: "status": "already probed" 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "osd": "2(4)", 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "status": "already probed" 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "osd": "4(2)", 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "status": "already probed" 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "recovery_progress": { 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "backfill_targets": [], 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "waiting_on_backfill": [], 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "last_backfill_started": "MIN", 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "backfill_info": { 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "begin": "MIN", 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "end": "MIN", 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "objects": [] 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "peer_backfill_info": [], 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "backfills_in_flight": [], 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "recovering": [], 2026-02-25T22:30:12.794 INFO:tasks.workunit.client.0.trial125.stdout: "pg_backend": { 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "recovery_ops": [], 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "read_ops": [] 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: { 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "name": "Started", 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "enter_time": "2026-02-25T22:29:57.075895+0000" 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: ], 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "scrubber": { 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "active": false, 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "must_scrub": false, 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "must_deep_scrub": false, 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "must_repair": false, 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "sched_time": "2026-02-27T07:41:57.977474+0000", 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "schedule": "not registered for scrubbing", 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "urgency": "periodic-regular", 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "shallow-target": { 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s0", 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "level": "shallow", 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "urgency": "periodic-regular", 2026-02-25T22:30:12.795 INFO:tasks.workunit.client.0.trial125.stdout: "sched_time": "2026-02-27T07:41:57.977474+0000", 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "orig_sched_time": "2026-02-27T07:41:57.977474+0000", 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "last_issue": "ok", 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "forced": false, 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "eligible": false, 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "queued": false, 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "active": false 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "deep-target": { 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "pgid": "2.0s0", 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "level": "deep", 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "urgency": "periodic-regular", 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "sched_time": "2026-03-05T11:10:36.670320+0000", 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "orig_sched_time": "2026-03-05T11:10:36.670320+0000", 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "last_issue": "ok", 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "forced": false, 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "eligible": false, 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "queued": false, 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "active": false 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: } 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: }, 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout: "agent_state": {} 2026-02-25T22:30:12.796 INFO:tasks.workunit.client.0.trial125.stdout:} 2026-02-25T22:30:12.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-25T22:30:12.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: grep -q obj75 2026-02-25T22:30:12.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-25T22:30:12.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2026-02-25T22:30:12.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: check=true 2026-02-25T22:30:12.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: test true == true 2026-02-25T22:30:12.918 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-25T22:30:12.918 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2026-02-25T22:30:12.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: check=0 2026-02-25T22:30:12.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:690: TEST_ec_recovery_unfound: test 0 == 0 2026-02-25T22:30:12.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2026-02-25T22:30:18.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:694: TEST_ec_recovery_unfound: test 124 = 124 2026-02-25T22:30:18.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:696: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2026-02-25T22:30:18.073 INFO:tasks.workunit.client.0.trial125.stderr:pg has no unfound objects 2026-02-25T22:30:18.080 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:698: TEST_ec_recovery_unfound: wait_for_clean 2026-02-25T22:30:18.080 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:30:18.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:30:18.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:30:18.081 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:30:18.082 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:18.082 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:18.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:18.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:18.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:18.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:30:18.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:30:18.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:30:18.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:30:18.194 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:30:18.194 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:30:18.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:30:18.402 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:30:18.402 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:30:18.402 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:30:18.402 INFO:tasks.workunit.client.0.trial125.stderr:4' 2026-02-25T22:30:18.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:30:18.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:18.403 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:30:18.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803791 2026-02-25T22:30:18.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803791 2026-02-25T22:30:18.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791' 2026-02-25T22:30:18.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:18.481 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:30:18.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607565 2026-02-25T22:30:18.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607565 2026-02-25T22:30:18.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-51539607565' 2026-02-25T22:30:18.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:18.561 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:30:18.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=206158430215 2026-02-25T22:30:18.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 206158430215 2026-02-25T22:30:18.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-51539607565 2-206158430215' 2026-02-25T22:30:18.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:18.639 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:30:18.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149708 2026-02-25T22:30:18.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149708 2026-02-25T22:30:18.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-51539607565 2-206158430215 3-111669149708' 2026-02-25T22:30:18.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:18.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:30:18.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888075 2026-02-25T22:30:18.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888075 2026-02-25T22:30:18.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-51539607565 2-206158430215 3-111669149708 4-146028888075' 2026-02-25T22:30:18.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:18.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803791 2026-02-25T22:30:18.798 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:18.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:30:18.799 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803791 2026-02-25T22:30:18.800 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:18.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803791 2026-02-25T22:30:18.801 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803791 2026-02-25T22:30:18.802 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803791' 2026-02-25T22:30:18.802 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:19.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803789 -lt 25769803791 2026-02-25T22:30:19.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:30:20.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:30:20.017 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:20.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803789 -lt 25769803791 2026-02-25T22:30:20.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:30:21.229 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:30:21.229 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:21.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803791 -lt 25769803791 2026-02-25T22:30:21.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:21.439 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607565 2026-02-25T22:30:21.439 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:21.440 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:30:21.441 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607565 2026-02-25T22:30:21.441 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:21.443 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607565 2026-02-25T22:30:21.443 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607565' 2026-02-25T22:30:21.443 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 51539607565 2026-02-25T22:30:21.443 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:30:21.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607566 -lt 51539607565 2026-02-25T22:30:21.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:21.655 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-206158430215 2026-02-25T22:30:21.655 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:21.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:30:21.657 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-206158430215 2026-02-25T22:30:21.657 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:21.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=206158430215 2026-02-25T22:30:21.658 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 206158430215 2026-02-25T22:30:21.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 206158430215' 2026-02-25T22:30:21.659 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:30:21.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 206158430215 -lt 206158430215 2026-02-25T22:30:21.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:21.877 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149708 2026-02-25T22:30:21.878 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:21.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:30:21.879 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149708 2026-02-25T22:30:21.879 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:21.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149708 2026-02-25T22:30:21.881 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149708 2026-02-25T22:30:21.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149708' 2026-02-25T22:30:21.882 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:30:22.097 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149709 -lt 111669149708 2026-02-25T22:30:22.097 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:22.098 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-146028888075 2026-02-25T22:30:22.098 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:22.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:30:22.100 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-146028888075 2026-02-25T22:30:22.101 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:22.102 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=146028888075 2026-02-25T22:30:22.102 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 146028888075' 2026-02-25T22:30:22.102 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 146028888075 2026-02-25T22:30:22.103 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:30:22.321 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 146028888076 -lt 146028888075 2026-02-25T22:30:22.322 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:30:22.322 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:22.323 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:22.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:30:22.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:30:22.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:30:22.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:30:22.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:30:22.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:30:22.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:30:22.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:30:22.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:30:22.816 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:30:22.816 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:22.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:23.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:30:23.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:30:23.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:30:23.091 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: seq 1 100 2026-02-25T22:30:23.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2026-02-25T22:30:23.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.113 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.113 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2026-02-25T22:30:23.113 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.133 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2026-02-25T22:30:23.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2026-02-25T22:30:23.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2026-02-25T22:30:23.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2026-02-25T22:30:23.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2026-02-25T22:30:23.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2026-02-25T22:30:23.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2026-02-25T22:30:23.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2026-02-25T22:30:23.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2026-02-25T22:30:23.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2026-02-25T22:30:23.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2026-02-25T22:30:23.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2026-02-25T22:30:23.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.395 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.395 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2026-02-25T22:30:23.395 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2026-02-25T22:30:23.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2026-02-25T22:30:23.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2026-02-25T22:30:23.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2026-02-25T22:30:23.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.510 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2026-02-25T22:30:23.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2026-02-25T22:30:23.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.554 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2026-02-25T22:30:23.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2026-02-25T22:30:23.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2026-02-25T22:30:23.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.624 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2026-02-25T22:30:23.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2026-02-25T22:30:23.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2026-02-25T22:30:23.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.687 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2026-02-25T22:30:23.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2026-02-25T22:30:23.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2026-02-25T22:30:23.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.750 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.751 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.751 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2026-02-25T22:30:23.751 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2026-02-25T22:30:23.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2026-02-25T22:30:23.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2026-02-25T22:30:23.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2026-02-25T22:30:23.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2026-02-25T22:30:23.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.893 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.893 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2026-02-25T22:30:23.893 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2026-02-25T22:30:23.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2026-02-25T22:30:23.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2026-02-25T22:30:23.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2026-02-25T22:30:23.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:23.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:23.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2026-02-25T22:30:23.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2026-02-25T22:30:24.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2026-02-25T22:30:24.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2026-02-25T22:30:24.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.089 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2026-02-25T22:30:24.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.113 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2026-02-25T22:30:24.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2026-02-25T22:30:24.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2026-02-25T22:30:24.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2026-02-25T22:30:24.182 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.203 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2026-02-25T22:30:24.204 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2026-02-25T22:30:24.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2026-02-25T22:30:24.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2026-02-25T22:30:24.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2026-02-25T22:30:24.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2026-02-25T22:30:24.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.324 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2026-02-25T22:30:24.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2026-02-25T22:30:24.346 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2026-02-25T22:30:24.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.386 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.387 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.387 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2026-02-25T22:30:24.387 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2026-02-25T22:30:24.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2026-02-25T22:30:24.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2026-02-25T22:30:24.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2026-02-25T22:30:24.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2026-02-25T22:30:24.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2026-02-25T22:30:24.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2026-02-25T22:30:24.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2026-02-25T22:30:24.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2026-02-25T22:30:24.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.605 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2026-02-25T22:30:24.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2026-02-25T22:30:24.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.649 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2026-02-25T22:30:24.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.669 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2026-02-25T22:30:24.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2026-02-25T22:30:24.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2026-02-25T22:30:24.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2026-02-25T22:30:24.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:704: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.750 INFO:tasks.workunit.client.0.trial125.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2026-02-25T22:30:24.752 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.752 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2026-02-25T22:30:24.752 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2026-02-25T22:30:24.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2026-02-25T22:30:24.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2026-02-25T22:30:24.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2026-02-25T22:30:24.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2026-02-25T22:30:24.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.893 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2026-02-25T22:30:24.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2026-02-25T22:30:24.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2026-02-25T22:30:24.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2026-02-25T22:30:24.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2026-02-25T22:30:24.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:24.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:24.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2026-02-25T22:30:24.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2026-02-25T22:30:25.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.027 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2026-02-25T22:30:25.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2026-02-25T22:30:25.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.077 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.079 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.079 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2026-02-25T22:30:25.079 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.102 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2026-02-25T22:30:25.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.126 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.126 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2026-02-25T22:30:25.126 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2026-02-25T22:30:25.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2026-02-25T22:30:25.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2026-02-25T22:30:25.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2026-02-25T22:30:25.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2026-02-25T22:30:25.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2026-02-25T22:30:25.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2026-02-25T22:30:25.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T22:30:25.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2026-02-25T22:30:25.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2026-02-25T22:30:25.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:711: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T22:30:25.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:713: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:30:25.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:30:25.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:30:25.586 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:30:25.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:30:25.866 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:30:25.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:30:25.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:30:25.876 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:25.876 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:30:25.876 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:25.876 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:25.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:25.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:25.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:25.988 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:25.988 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:25.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:25.990 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:25.991 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:25.992 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:25.992 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:25.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:25.993 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:25.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:25.994 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:25.995 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:25.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:25.996 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:30:26.021 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:26.022 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:26.022 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:26.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:30:26.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:26.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:26.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:30:26.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:30:26.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:30:26.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:30:26.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:30:26.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:26.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:30:26.025 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:26.025 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:26.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:26.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:26.025 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:26.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:26.027 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:26.027 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:26.028 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:26.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:26.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:26.030 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:26.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:26.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:26.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:26.032 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:26.033 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:26.034 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:26.034 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:30:26.035 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:26.036 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:26.036 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:26.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:30:26.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:26.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:26.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:30:26.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:30:26.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:26.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:26.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:30:26.040 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:30:26.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:30:26.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:30:26.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:30:26.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:30:26.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:30:26.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:30:26.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:30:26.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:30:26.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:30:26.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:30:26.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:30:26.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:26.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:26.064 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:26.064 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:26.064 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:26.065 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:26.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:30:26.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:30:26.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:30:26.096 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:26.096 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:26.096 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:30:26.097 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:30:26.097 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:26.097 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:26.097 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:26.097 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:26.097 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:26.097 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:26.098 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:30:26.098 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:26.098 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:30:26.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:30:26.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:26.154 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:26.154 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:30:26.155 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:30:26.155 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:26.155 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:26.155 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:26.155 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:26.155 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:26.155 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:26.156 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:30:26.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:26.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:30:26.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:30:26.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:30:26.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:30:26.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:30:26.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:30:26.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:30:26.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:30:26.318 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:30:26.318 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:26.318 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:26.319 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:26.319 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:26.319 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:26.319 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:26.320 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:26.320 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:26.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:30:26.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:30:26.458 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:30:26.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:30:27.469 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:30:27.469 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:27.469 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:27.469 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:27.469 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:27.470 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:27.470 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:30:27.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:30:27.470 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:30:27.518 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:30:27.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:30:27.525 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:30:26.081+0000 7f0a08701ec0 0 load: jerasure load: lrc 2026-02-25T22:30:27.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2026-02-25T22:30:27.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:410: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2026-02-25T22:30:27.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: local objname=myobject 2026-02-25T22:30:27.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: setup_osds 7 2026-02-25T22:30:27.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-25T22:30:27.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:30:27.527 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-25T22:30:27.528 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-25T22:30:27.531 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:30:27.531 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:30:27.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:30:27.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:27.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:30:27.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:27.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:30:27.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:30:27.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:27.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:27.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:27.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:30:27.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:27.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:27.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:27.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:30:27.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:30:27.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:27.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:27.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:27.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:27.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:27.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:27.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:30:27.535 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:27.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5698eb31-b4b6-442d-82db-deec78768352 2026-02-25T22:30:27.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5698eb31-b4b6-442d-82db-deec78768352' 2026-02-25T22:30:27.535 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 5698eb31-b4b6-442d-82db-deec78768352 2026-02-25T22:30:27.536 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:27.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQADeJ9p78doIBAABr8DE7KE+QbYPwnDFoUDLA== 2026-02-25T22:30:27.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQADeJ9p78doIBAABr8DE7KE+QbYPwnDFoUDLA=="}' 2026-02-25T22:30:27.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5698eb31-b4b6-442d-82db-deec78768352 -i td/test-erasure-eio/0/new.json 2026-02-25T22:30:27.656 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:30:27.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:30:27.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADeJ9p78doIBAABr8DE7KE+QbYPwnDFoUDLA== --osd-uuid 5698eb31-b4b6-442d-82db-deec78768352 2026-02-25T22:30:27.680 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:27.677+0000 7f90e3304a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:27.681 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:27.677+0000 7f90e3304a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:27.682 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:27.677+0000 7f90e3304a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:27.683 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:27.677+0000 7f90e3304a40 -1 bdev(0x557439960400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:27.683 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:27.677+0000 7f90e3304a40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:30:28.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:30:28.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:28.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:30:28.375 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:30:28.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:28.521 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:30:28.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:30:28.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:28.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:28.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:28.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:28.536 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:28.533+0000 7f8939738a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:28.538 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:28.533+0000 7f8939738a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:28.539 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:28.533+0000 7f8939738a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:28.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:30:28.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:28.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:30:28.627 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:28.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:28.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:28.628 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:30:28.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:28.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:28.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:28.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:28.917 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:28.913+0000 7f8939738a40 -1 Falling back to public interface 2026-02-25T22:30:29.541 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:29.537+0000 7f8939738a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:30:29.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:29.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:29.843 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:30:29.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:29.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:29.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:30.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:31.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:31.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:31.054 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:30:31.055 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:31.055 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:31.055 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:31.264 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/202406226,v1:127.0.0.1:6803/202406226] [v2:127.0.0.1:6804/202406226,v1:127.0.0.1:6805/202406226] exists,up 5698eb31-b4b6-442d-82db-deec78768352 2026-02-25T22:30:31.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:31.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:31.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:31.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:30:31.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:30:31.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:30:31.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:31.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:30:31.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:31.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:31.269 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:31.270 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:31.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:31.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:31.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:30:31.271 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:31.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f58a40b4-72db-4645-9f10-c5cc73092bf4 2026-02-25T22:30:31.272 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 f58a40b4-72db-4645-9f10-c5cc73092bf4 2026-02-25T22:30:31.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f58a40b4-72db-4645-9f10-c5cc73092bf4' 2026-02-25T22:30:31.272 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:31.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAHeJ9pAce/EBAA0zCorndp5KXjHMB7q9Q/Xw== 2026-02-25T22:30:31.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAHeJ9pAce/EBAA0zCorndp5KXjHMB7q9Q/Xw=="}' 2026-02-25T22:30:31.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f58a40b4-72db-4645-9f10-c5cc73092bf4 -i td/test-erasure-eio/1/new.json 2026-02-25T22:30:31.505 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:30:31.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:30:31.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHeJ9pAce/EBAA0zCorndp5KXjHMB7q9Q/Xw== --osd-uuid f58a40b4-72db-4645-9f10-c5cc73092bf4 2026-02-25T22:30:31.532 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:31.529+0000 7f22f685fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:31.534 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:31.529+0000 7f22f685fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:31.535 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:31.529+0000 7f22f685fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:31.535 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:31.533+0000 7f22f685fa40 -1 bdev(0x56337235c400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:31.535 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:31.533+0000 7f22f685fa40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:30:32.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:30:32.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:32.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:30:32.176 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:30:32.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:32.472 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:30:32.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:30:32.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:32.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:32.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:32.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:32.496 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:32.493+0000 7f5eac0c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:32.498 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:32.493+0000 7f5eac0c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:32.499 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:32.493+0000 7f5eac0c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:32.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:30:32.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:32.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:30:32.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:32.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:32.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:32.684 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:30:32.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:32.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:32.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:30:32.895 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:32.985 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:32.981+0000 7f5eac0c9a40 -1 Falling back to public interface 2026-02-25T22:30:33.608 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:33.605+0000 7f5eac0c9a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:30:33.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:33.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:33.897 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:30:33.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:33.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:33.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:30:34.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:35.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:35.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:35.119 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:30:35.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:35.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:35.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:30:35.328 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4107612300,v1:127.0.0.1:6811/4107612300] [v2:127.0.0.1:6812/4107612300,v1:127.0.0.1:6813/4107612300] exists,up f58a40b4-72db-4645-9f10-c5cc73092bf4 2026-02-25T22:30:35.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:35.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:35.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:35.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:30:35.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:30:35.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:35.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:35.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:30:35.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:30:35.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:35.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:35.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:35.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:35.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:35.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:35.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:30:35.335 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:35.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b81e7649-4fca-4730-9ebf-f8ccc0fd44b7 2026-02-25T22:30:35.335 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 b81e7649-4fca-4730-9ebf-f8ccc0fd44b7 2026-02-25T22:30:35.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 b81e7649-4fca-4730-9ebf-f8ccc0fd44b7' 2026-02-25T22:30:35.336 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:35.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQALeJ9p7WiCFBAAHrSZZ9kzBXllM3YJwsl+dQ== 2026-02-25T22:30:35.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQALeJ9p7WiCFBAAHrSZZ9kzBXllM3YJwsl+dQ=="}' 2026-02-25T22:30:35.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b81e7649-4fca-4730-9ebf-f8ccc0fd44b7 -i td/test-erasure-eio/2/new.json 2026-02-25T22:30:35.567 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:30:35.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:30:35.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALeJ9p7WiCFBAAHrSZZ9kzBXllM3YJwsl+dQ== --osd-uuid b81e7649-4fca-4730-9ebf-f8ccc0fd44b7 2026-02-25T22:30:35.594 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:35.589+0000 7f7cc596ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:35.595 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:35.593+0000 7f7cc596ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:35.596 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:35.593+0000 7f7cc596ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:35.597 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:35.593+0000 7f7cc596ba40 -1 bdev(0x563effb6e400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:35.597 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:35.593+0000 7f7cc596ba40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:30:36.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:30:36.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:36.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:30:36.243 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:30:36.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:36.524 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:30:36.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:30:36.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:36.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:36.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:36.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:36.541 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:36.537+0000 7f084192ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:36.542 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:36.537+0000 7f084192ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:36.543 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:36.537+0000 7f084192ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:36.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:30:36.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:36.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:30:36.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:36.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:36.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:36.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:36.730 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:30:36.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:36.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:30:36.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:37.013 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:37.009+0000 7f084192ca40 -1 Falling back to public interface 2026-02-25T22:30:37.589 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:37.585+0000 7f084192ca40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:30:37.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:37.943 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:37.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:37.944 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:30:37.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:37.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:30:38.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:39.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:39.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:39.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:39.150 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:30:39.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:39.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:30:39.364 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2924581846,v1:127.0.0.1:6819/2924581846] [v2:127.0.0.1:6820/2924581846,v1:127.0.0.1:6821/2924581846] exists,up b81e7649-4fca-4730-9ebf-f8ccc0fd44b7 2026-02-25T22:30:39.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:39.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:39.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:39.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:30:39.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:30:39.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:30:39.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:39.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:30:39.369 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:30:39.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:39.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:39.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:39.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:39.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:39.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:39.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:30:39.372 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:39.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9c7335d7-4216-4184-80ef-425820113fd2 2026-02-25T22:30:39.372 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 9c7335d7-4216-4184-80ef-425820113fd2 2026-02-25T22:30:39.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 9c7335d7-4216-4184-80ef-425820113fd2' 2026-02-25T22:30:39.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:39.382 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAPeJ9p41u7FhAAhZ45IG2QSgVNQmj6b6ALKQ== 2026-02-25T22:30:39.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAPeJ9p41u7FhAAhZ45IG2QSgVNQmj6b6ALKQ=="}' 2026-02-25T22:30:39.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9c7335d7-4216-4184-80ef-425820113fd2 -i td/test-erasure-eio/3/new.json 2026-02-25T22:30:39.597 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:30:39.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:30:39.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPeJ9p41u7FhAAhZ45IG2QSgVNQmj6b6ALKQ== --osd-uuid 9c7335d7-4216-4184-80ef-425820113fd2 2026-02-25T22:30:39.626 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:39.621+0000 7fa5b20f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:39.628 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:39.625+0000 7fa5b20f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:39.629 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:39.625+0000 7fa5b20f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:39.630 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:39.625+0000 7fa5b20f8a40 -1 bdev(0x55def07dc400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:39.630 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:39.625+0000 7fa5b20f8a40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:30:40.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:30:40.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:40.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:30:40.283 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:30:40.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:40.573 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:30:40.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:30:40.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:40.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:40.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:40.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:40.597 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:40.593+0000 7f7bc7fcca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:40.598 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:40.593+0000 7f7bc7fcca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:40.599 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:40.597+0000 7f7bc7fcca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:40.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:30:40.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:40.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:30:40.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:40.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:40.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:40.778 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:30:40.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:40.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:40.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:30:40.957 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:40.953+0000 7f7bc7fcca40 -1 Falling back to public interface 2026-02-25T22:30:40.987 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:41.587 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:41.581+0000 7f7bc7fcca40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:30:41.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:41.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:41.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:41.990 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:30:41.990 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:41.991 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:30:42.197 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:43.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:43.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:43.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:43.199 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:30:43.200 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:43.200 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:30:43.410 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2809454074,v1:127.0.0.1:6827/2809454074] [v2:127.0.0.1:6828/2809454074,v1:127.0.0.1:6829/2809454074] exists,up 9c7335d7-4216-4184-80ef-425820113fd2 2026-02-25T22:30:43.414 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:43.414 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:43.414 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:43.414 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:30:43.414 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T22:30:43.415 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:43.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:43.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:30:43.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:30:43.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:43.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:43.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:43.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:43.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:43.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:43.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T22:30:43.418 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:43.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=618758ca-10b0-47d5-8a0f-4e7f432e217b 2026-02-25T22:30:43.418 INFO:tasks.workunit.client.0.trial125.stdout:add osd4 618758ca-10b0-47d5-8a0f-4e7f432e217b 2026-02-25T22:30:43.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 618758ca-10b0-47d5-8a0f-4e7f432e217b' 2026-02-25T22:30:43.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:43.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQATeJ9pVLlmGRAA13WUc5gbVQ811V/2iVvSMw== 2026-02-25T22:30:43.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQATeJ9pVLlmGRAA13WUc5gbVQ811V/2iVvSMw=="}' 2026-02-25T22:30:43.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 618758ca-10b0-47d5-8a0f-4e7f432e217b -i td/test-erasure-eio/4/new.json 2026-02-25T22:30:43.640 INFO:tasks.workunit.client.0.trial125.stdout:4 2026-02-25T22:30:43.649 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T22:30:43.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATeJ9pVLlmGRAA13WUc5gbVQ811V/2iVvSMw== --osd-uuid 618758ca-10b0-47d5-8a0f-4e7f432e217b 2026-02-25T22:30:43.666 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:43.661+0000 7f5d3ef65a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:43.667 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:43.665+0000 7f5d3ef65a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:43.668 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:43.665+0000 7f5d3ef65a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:43.669 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:43.665+0000 7f5d3ef65a40 -1 bdev(0x5567cf8e0400 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:43.669 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:43.665+0000 7f5d3ef65a40 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T22:30:44.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T22:30:44.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:44.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T22:30:44.315 INFO:tasks.workunit.client.0.trial125.stdout:adding osd4 key to auth repository 2026-02-25T22:30:44.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:44.605 INFO:tasks.workunit.client.0.trial125.stdout:start osd.4 2026-02-25T22:30:44.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T22:30:44.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:44.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:44.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:44.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:44.628 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:44.625+0000 7fe3b114ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:44.630 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:44.625+0000 7fe3b114ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:44.631 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:44.625+0000 7fe3b114ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:44.817 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T22:30:44.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:44.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T22:30:44.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:44.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:44.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:44.818 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:30:44.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:44.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:44.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:30:45.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:45.073 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:45.069+0000 7fe3b114ea40 -1 Falling back to public interface 2026-02-25T22:30:45.712 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:45.709+0000 7fe3b114ea40 -1 osd.4 0 log_to_monitors true 2026-02-25T22:30:46.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:46.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:46.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:46.032 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:30:46.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:46.033 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:30:46.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:47.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:47.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:47.246 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:30:47.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:47.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:47.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T22:30:47.457 INFO:tasks.workunit.client.0.trial125.stdout:osd.4 up in weight 1 up_from 33 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1637114199,v1:127.0.0.1:6835/1637114199] [v2:127.0.0.1:6836/1637114199,v1:127.0.0.1:6837/1637114199] exists,up 618758ca-10b0-47d5-8a0f-4e7f432e217b 2026-02-25T22:30:47.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:47.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:47.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:47.462 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:47.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-25T22:30:47.464 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:47.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1b5137db-770b-4000-be55-d4c9f19ea4d3 2026-02-25T22:30:47.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 1b5137db-770b-4000-be55-d4c9f19ea4d3' 2026-02-25T22:30:47.465 INFO:tasks.workunit.client.0.trial125.stdout:add osd5 1b5137db-770b-4000-be55-d4c9f19ea4d3 2026-02-25T22:30:47.465 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:47.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAXeJ9pUSYzHBAAToUH/I5yACAUlCuMJmxzoA== 2026-02-25T22:30:47.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAXeJ9pUSYzHBAAToUH/I5yACAUlCuMJmxzoA=="}' 2026-02-25T22:30:47.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1b5137db-770b-4000-be55-d4c9f19ea4d3 -i td/test-erasure-eio/5/new.json 2026-02-25T22:30:47.690 INFO:tasks.workunit.client.0.trial125.stdout:5 2026-02-25T22:30:47.699 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-25T22:30:47.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAXeJ9pUSYzHBAAToUH/I5yACAUlCuMJmxzoA== --osd-uuid 1b5137db-770b-4000-be55-d4c9f19ea4d3 2026-02-25T22:30:47.716 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:47.713+0000 7f58da8fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:47.718 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:47.713+0000 7f58da8fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:47.719 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:47.713+0000 7f58da8fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:47.719 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:47.713+0000 7f58da8fda40 -1 bdev(0x561e7a60c400 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:47.719 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:47.717+0000 7f58da8fda40 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-25T22:30:48.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-25T22:30:48.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:48.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T22:30:48.391 INFO:tasks.workunit.client.0.trial125.stdout:adding osd5 key to auth repository 2026-02-25T22:30:48.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:48.672 INFO:tasks.workunit.client.0.trial125.stdout:start osd.5 2026-02-25T22:30:48.672 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T22:30:48.672 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:48.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:48.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:48.673 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:48.688 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:48.685+0000 7f14e74c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:48.690 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:48.685+0000 7f14e74c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:48.691 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:48.685+0000 7f14e74c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:48.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T22:30:48.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:48.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T22:30:48.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:48.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:48.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:48.883 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:30:48.883 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:48.883 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:48.883 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:30:49.077 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:49.073+0000 7f14e74c5a40 -1 Falling back to public interface 2026-02-25T22:30:49.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:49.740 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:49.737+0000 7f14e74c5a40 -1 osd.5 0 log_to_monitors true 2026-02-25T22:30:50.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:50.092 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:50.093 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:30:50.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:50.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:50.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:30:50.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:51.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:51.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:51.299 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:30:51.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:51.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:51.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T22:30:51.509 INFO:tasks.workunit.client.0.trial125.stdout:osd.5 up in weight 1 up_from 41 up_thru 42 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1525309263,v1:127.0.0.1:6843/1525309263] [v2:127.0.0.1:6844/1525309263,v1:127.0.0.1:6845/1525309263] exists,up 1b5137db-770b-4000-be55-d4c9f19ea4d3 2026-02-25T22:30:51.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:51.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:51.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:51.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:30:51.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-25T22:30:51.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:51.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:51.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:30:51.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:30:51.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:51.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:51.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:51.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:51.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:51.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:51.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-25T22:30:51.517 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:51.517 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8ace26a9-5d6b-4221-abfa-501d95637a72 2026-02-25T22:30:51.517 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 8ace26a9-5d6b-4221-abfa-501d95637a72' 2026-02-25T22:30:51.517 INFO:tasks.workunit.client.0.trial125.stdout:add osd6 8ace26a9-5d6b-4221-abfa-501d95637a72 2026-02-25T22:30:51.518 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:51.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAbeJ9pHwJWHxAA2BtgOKyvSzhgUYw2h7sGrw== 2026-02-25T22:30:51.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAbeJ9pHwJWHxAA2BtgOKyvSzhgUYw2h7sGrw=="}' 2026-02-25T22:30:51.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8ace26a9-5d6b-4221-abfa-501d95637a72 -i td/test-erasure-eio/6/new.json 2026-02-25T22:30:51.745 INFO:tasks.workunit.client.0.trial125.stdout:6 2026-02-25T22:30:51.755 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-25T22:30:51.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbeJ9pHwJWHxAA2BtgOKyvSzhgUYw2h7sGrw== --osd-uuid 8ace26a9-5d6b-4221-abfa-501d95637a72 2026-02-25T22:30:51.771 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:51.765+0000 7f3e3044ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:51.772 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:51.769+0000 7f3e3044ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:51.773 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:51.769+0000 7f3e3044ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:51.774 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:51.769+0000 7f3e3044ea40 -1 bdev(0x55cb17a84400 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:51.774 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:51.769+0000 7f3e3044ea40 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-25T22:30:52.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-25T22:30:52.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:52.371 INFO:tasks.workunit.client.0.trial125.stdout:adding osd6 key to auth repository 2026-02-25T22:30:52.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T22:30:52.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:52.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T22:30:52.663 INFO:tasks.workunit.client.0.trial125.stdout:start osd.6 2026-02-25T22:30:52.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:52.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:52.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:52.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:52.686 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:52.681+0000 7fdd4d6c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:52.688 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:52.685+0000 7fdd4d6c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:52.689 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:52.685+0000 7fdd4d6c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:52.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T22:30:52.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:52.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T22:30:52.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:52.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:52.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:52.882 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:30:52.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:52.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:52.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:30:53.093 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:53.137 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:53.133+0000 7fdd4d6c1a40 -1 Falling back to public interface 2026-02-25T22:30:53.720 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:30:53.717+0000 7fdd4d6c1a40 -1 osd.6 0 log_to_monitors true 2026-02-25T22:30:54.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:54.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:54.095 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:30:54.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:54.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:54.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:30:54.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:55.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:55.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:55.312 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:30:55.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:55.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:55.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T22:30:55.525 INFO:tasks.workunit.client.0.trial125.stdout:osd.6 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/684012916,v1:127.0.0.1:6851/684012916] [v2:127.0.0.1:6852/684012916,v1:127.0.0.1:6853/684012916] exists,up 8ace26a9-5d6b-4221-abfa-501d95637a72 2026-02-25T22:30:55.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:55.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:55.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:55.526 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:30:55.526 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:30:55.526 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:30:55.527 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:55.527 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:55.527 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:30:55.527 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:30:55.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:30:55.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:30:55.592 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:30:55.598 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:30:55.601 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:30:28.917+0000 7f8939738a40 0 load: jerasure load: lrc 2026-02-25T22:30:55.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:415: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2026-02-25T22:30:55.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:416: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T22:30:55.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:30:55.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:30:55.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T22:30:55.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:30:55.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T22:30:55.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:30:55.603 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T22:30:55.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:30:55.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:30:56.242 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:30:56.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:30:57.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:30:57.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:30:57.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:30:57.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:30:57.255 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:30:57.255 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:57.255 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:57.256 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:57.256 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:57.256 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:57.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:30:57.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:30:57.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:30:57.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:30:57.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:30:57.358 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:30:57.563 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:30:57.564 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:30:57.564 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:30:57.564 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:30:57.564 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:30:57.564 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:30:57.564 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:30:57.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:30:57.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:57.565 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:30:57.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803784 2026-02-25T22:30:57.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803784 2026-02-25T22:30:57.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784' 2026-02-25T22:30:57.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:57.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:30:57.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574855 2026-02-25T22:30:57.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574855 2026-02-25T22:30:57.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-55834574855' 2026-02-25T22:30:57.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:57.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:30:57.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411334 2026-02-25T22:30:57.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411334 2026-02-25T22:30:57.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-55834574855 2-77309411334' 2026-02-25T22:30:57.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:57.801 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:30:57.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182406 2026-02-25T22:30:57.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182406 2026-02-25T22:30:57.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-55834574855 2-77309411334 3-107374182406' 2026-02-25T22:30:57.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:57.881 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:30:57.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920774 2026-02-25T22:30:57.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920774 2026-02-25T22:30:57.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-55834574855 2-77309411334 3-107374182406 4-141733920774' 2026-02-25T22:30:57.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:57.960 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:30:58.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659140 2026-02-25T22:30:58.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659140 2026-02-25T22:30:58.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-55834574855 2-77309411334 3-107374182406 4-141733920774 5-176093659140' 2026-02-25T22:30:58.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:58.037 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:30:58.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397507 2026-02-25T22:30:58.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397507 2026-02-25T22:30:58.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-55834574855 2-77309411334 3-107374182406 4-141733920774 5-176093659140 6-210453397507' 2026-02-25T22:30:58.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:58.116 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803784 2026-02-25T22:30:58.116 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:58.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:30:58.118 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803784 2026-02-25T22:30:58.118 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:58.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803784 2026-02-25T22:30:58.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-25T22:30:58.120 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803784 2026-02-25T22:30:58.121 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:58.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803784 2026-02-25T22:30:58.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:30:59.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:30:59.329 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:59.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803784 -lt 25769803784 2026-02-25T22:30:59.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:59.544 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574855 2026-02-25T22:30:59.544 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:59.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:30:59.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574855 2026-02-25T22:30:59.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:59.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574855 2026-02-25T22:30:59.548 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574855 2026-02-25T22:30:59.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574855' 2026-02-25T22:30:59.549 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:30:59.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574855 -lt 55834574855 2026-02-25T22:30:59.759 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:59.760 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411334 2026-02-25T22:30:59.760 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:59.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:30:59.762 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411334 2026-02-25T22:30:59.763 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:59.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411334 2026-02-25T22:30:59.764 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 77309411334 2026-02-25T22:30:59.765 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411334' 2026-02-25T22:30:59.765 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:30:59.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411334 -lt 77309411334 2026-02-25T22:30:59.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:59.978 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-107374182406 2026-02-25T22:30:59.978 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:59.980 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:30:59.980 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-107374182406 2026-02-25T22:30:59.981 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:59.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=107374182406 2026-02-25T22:30:59.982 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 107374182406 2026-02-25T22:30:59.983 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 107374182406' 2026-02-25T22:30:59.983 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:31:00.206 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 107374182406 -lt 107374182406 2026-02-25T22:31:00.206 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:00.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-141733920774 2026-02-25T22:31:00.207 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:00.208 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:31:00.209 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-141733920774 2026-02-25T22:31:00.209 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:00.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=141733920774 2026-02-25T22:31:00.211 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 141733920774 2026-02-25T22:31:00.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 141733920774' 2026-02-25T22:31:00.211 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:31:00.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 141733920774 -lt 141733920774 2026-02-25T22:31:00.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:00.433 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-176093659140 2026-02-25T22:31:00.433 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:00.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:31:00.435 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-176093659140 2026-02-25T22:31:00.435 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:00.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=176093659140 2026-02-25T22:31:00.437 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 176093659140 2026-02-25T22:31:00.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 176093659140' 2026-02-25T22:31:00.437 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:31:00.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 176093659140 -lt 176093659140 2026-02-25T22:31:00.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:00.651 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-210453397507 2026-02-25T22:31:00.651 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:00.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T22:31:00.653 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-210453397507 2026-02-25T22:31:00.653 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:00.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397507 2026-02-25T22:31:00.655 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.6 seq 210453397507 2026-02-25T22:31:00.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 210453397507' 2026-02-25T22:31:00.655 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T22:31:00.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397508 -lt 210453397507 2026-02-25T22:31:00.874 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:31:00.874 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:00.875 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:01.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:31:01.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:31:01.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:31:01.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:31:01.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:31:01.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:31:01.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:31:01.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:31:01.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:31:01.358 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:31:01.358 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:01.359 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:01.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:31:01.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:31:01.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:31:01.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-25T22:31:01.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:31:01.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:31:01.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-25T22:31:01.629 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:01.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:31:01.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:01.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:31:01.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:01.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:31:01.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:01.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:31:01.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-25T22:31:01.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2026-02-25T22:31:01.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:31:01.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:31:01.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:31:01.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:31:01.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:31:01.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:31:01.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T22:31:01.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:31:01.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:31:01.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:31:01.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T22:31:01.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:31:01.651 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T22:31:01.651 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:31:01.651 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T22:31:01.652 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:31:01.652 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:31:01.864 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:31:01.865 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:31:01.865 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:01.865 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:31:01.865 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:31:01.865 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2026-02-25T22:31:01.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2026-02-25T22:31:01.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:31:01.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T22:31:01.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:31:01.866 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T22:31:01.867 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:31:01.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T22:31:01.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:31:01.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T22:31:01.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:31:01.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:31:01.868 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:31:01.868 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T22:31:01.868 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:31:01.868 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:31:01.868 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:01.869 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:01.869 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:01.869 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:31:01.869 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:31:01.928 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:31:01.928 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:31:01.928 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:31:01.929 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T22:31:01.929 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:31:01.929 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:31:01.930 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:01.930 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:01.930 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:01.930 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:31:01.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:31:01.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2026-02-25T22:31:01.988 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2026-02-25T22:31:01.988 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:31:01.988 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T22:31:01.989 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T22:31:01.989 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:31:02.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:31:02.200 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:31:02.200 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:02.200 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:31:02.201 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:31:02.201 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2026-02-25T22:31:02.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: initial_osds=('3' '5' '1' '0' '6') 2026-02-25T22:31:02.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: local -a initial_osds 2026-02-25T22:31:02.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:422: TEST_ec_single_recovery_error: local last_osd=6 2026-02-25T22:31:02.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2026-02-25T22:31:02.202 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:02.202 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:02.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:02.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:02.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:02.510 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:02.510 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:425: TEST_ec_single_recovery_error: ceph osd down 6 2026-02-25T22:31:02.724 INFO:tasks.workunit.client.0.trial125.stderr:osd.6 is already down. 2026-02-25T22:31:02.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_single_recovery_error: ceph osd out 6 2026-02-25T22:31:02.951 INFO:tasks.workunit.client.0.trial125.stderr:osd.6 is already out. 2026-02-25T22:31:02.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:429: TEST_ec_single_recovery_error: wait_for_clean 2026-02-25T22:31:02.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:31:02.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:31:02.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:31:02.961 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:31:02.962 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:02.962 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:02.962 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:02.963 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:02.963 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:03.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:31:03.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:31:03.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:31:03.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:31:03.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:03.076 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:03.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:31:03.290 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:03.290 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:31:03.290 INFO:tasks.workunit.client.0.trial125.stderr:3 2026-02-25T22:31:03.290 INFO:tasks.workunit.client.0.trial125.stderr:4 2026-02-25T22:31:03.290 INFO:tasks.workunit.client.0.trial125.stderr:5 2026-02-25T22:31:03.290 INFO:tasks.workunit.client.0.trial125.stderr:6' 2026-02-25T22:31:03.290 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:03.290 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:03.291 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:03.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803787 2026-02-25T22:31:03.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803787 2026-02-25T22:31:03.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787' 2026-02-25T22:31:03.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:03.369 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:31:03.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574858 2026-02-25T22:31:03.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574858 2026-02-25T22:31:03.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-55834574858' 2026-02-25T22:31:03.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:03.447 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:31:03.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411337 2026-02-25T22:31:03.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411337 2026-02-25T22:31:03.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-77309411337' 2026-02-25T22:31:03.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:03.525 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:31:03.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182409 2026-02-25T22:31:03.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182409 2026-02-25T22:31:03.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-77309411337 3-107374182409' 2026-02-25T22:31:03.604 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:03.604 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T22:31:03.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920777 2026-02-25T22:31:03.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920777 2026-02-25T22:31:03.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-77309411337 3-107374182409 4-141733920777' 2026-02-25T22:31:03.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:03.682 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T22:31:03.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659143 2026-02-25T22:31:03.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659143 2026-02-25T22:31:03.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-77309411337 3-107374182409 4-141733920777 5-176093659143' 2026-02-25T22:31:03.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:03.761 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T22:31:03.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq= 2026-02-25T22:31:03.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: true 2026-02-25T22:31:03.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z '' 2026-02-25T22:31:03.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: continue 2026-02-25T22:31:03.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:03.821 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803787 2026-02-25T22:31:03.821 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:03.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:03.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803787 2026-02-25T22:31:03.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:03.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803787 2026-02-25T22:31:03.824 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803787' 2026-02-25T22:31:03.825 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803787 2026-02-25T22:31:03.825 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:04.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803787 2026-02-25T22:31:04.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:05.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:31:05.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:05.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803787 -lt 25769803787 2026-02-25T22:31:05.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:05.245 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574858 2026-02-25T22:31:05.245 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:05.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:31:05.247 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574858 2026-02-25T22:31:05.247 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:05.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574858 2026-02-25T22:31:05.249 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574858 2026-02-25T22:31:05.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574858' 2026-02-25T22:31:05.249 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:31:05.472 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574858 -lt 55834574858 2026-02-25T22:31:05.472 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:05.473 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411337 2026-02-25T22:31:05.473 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:05.474 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:31:05.475 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411337 2026-02-25T22:31:05.475 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:05.476 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411337 2026-02-25T22:31:05.477 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 77309411337 2026-02-25T22:31:05.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411337' 2026-02-25T22:31:05.477 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:31:05.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411338 -lt 77309411337 2026-02-25T22:31:05.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:05.702 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-107374182409 2026-02-25T22:31:05.702 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:05.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:31:05.704 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-107374182409 2026-02-25T22:31:05.705 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:05.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=107374182409 2026-02-25T22:31:05.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 107374182409' 2026-02-25T22:31:05.706 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 107374182409 2026-02-25T22:31:05.707 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:31:05.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 107374182409 -lt 107374182409 2026-02-25T22:31:05.926 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:05.926 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-141733920777 2026-02-25T22:31:05.926 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:05.928 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T22:31:05.928 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-141733920777 2026-02-25T22:31:05.929 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:05.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=141733920777 2026-02-25T22:31:05.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 141733920777' 2026-02-25T22:31:05.930 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.4 seq 141733920777 2026-02-25T22:31:05.931 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T22:31:06.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 141733920777 -lt 141733920777 2026-02-25T22:31:06.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:06.151 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-176093659143 2026-02-25T22:31:06.151 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:06.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T22:31:06.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-176093659143 2026-02-25T22:31:06.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:06.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=176093659143 2026-02-25T22:31:06.155 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.5 seq 176093659143 2026-02-25T22:31:06.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 176093659143' 2026-02-25T22:31:06.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T22:31:06.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 176093659143 -lt 176093659143 2026-02-25T22:31:06.377 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:31:06.378 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:06.378 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:06.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:31:06.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:31:06.653 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:31:06.653 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:31:06.653 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:31:06.654 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:31:06.654 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:31:06.654 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:31:06.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:31:06.858 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:31:06.858 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:06.858 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:07.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:31:07.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:31:07.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:31:07.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2026-02-25T22:31:07.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:31:07.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:31:07.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2026-02-25T22:31:07.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:31:07.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:31:07.132 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2026-02-25T22:31:07.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:31:07.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:31:07.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:31:07.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:31:07.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:31:07.416 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:31:07.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:31:07.701 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:31:07.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:31:07.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:31:07.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:31:07.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:31:07.712 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:07.712 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:07.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:07.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:07.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:08.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:08.024 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:31:08.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:31:08.026 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:31:08.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:31:08.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:31:08.029 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:31:08.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:08.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:31:08.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:31:08.031 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:08.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:31:08.033 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:31:08.033 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:31:08.053 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:31:08.053 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.053 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:08.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:31:08.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:31:08.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:31:08.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:31:08.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:31:08.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:31:08.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:31:08.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:31:08.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:31:08.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:31:08.056 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:08.056 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:08.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:08.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:08.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:08.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:08.059 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:31:08.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:31:08.060 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:31:08.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:31:08.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:31:08.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:31:08.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:08.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:31:08.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:31:08.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:08.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:31:08.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:31:08.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:31:08.068 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:31:08.068 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.068 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:08.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:31:08.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:31:08.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:31:08.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:31:08.071 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:31:08.071 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.071 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:08.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:31:08.072 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:31:08.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:31:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:31:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:31:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:31:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:31:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:31:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:31:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:31:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:31:08.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:31:08.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:31:08.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:08.095 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:08.096 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:08.096 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.096 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:08.096 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:08.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:31:08.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:31:08.124 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:31:08.124 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:31:08.124 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:31:08.125 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:31:08.126 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:31:08.126 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:31:08.126 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:31:08.126 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:31:08.126 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:08.126 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.126 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:08.126 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:31:08.127 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:31:08.127 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:31:08.182 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:31:08.182 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:31:08.183 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:31:08.183 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:31:08.183 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:31:08.183 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:31:08.184 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:31:08.184 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:31:08.184 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:08.185 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.185 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:08.185 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:31:08.185 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:31:08.185 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:31:08.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:31:08.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:31:08.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:31:08.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:31:08.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:31:08.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:31:08.241 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:31:08.353 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:31:08.353 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:08.353 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:08.354 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:08.354 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.354 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:08.354 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:08.354 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:31:08.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:31:08.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:31:08.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:31:08.478 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:31:08.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:31:09.489 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:31:09.489 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:31:09.489 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:31:09.489 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:09.489 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:09.489 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:09.490 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:31:09.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:31:09.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:31:09.540 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:31:09.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:31:09.547 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:31:08.109+0000 7f6664832ec0 0 load: jerasure load: lrc 2026-02-25T22:31:09.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2026-02-25T22:31:09.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2026-02-25T22:31:09.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_0: setup_osds 4 2026-02-25T22:31:09.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T22:31:09.549 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:31:09.549 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T22:31:09.550 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T22:31:09.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:31:09.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:31:09.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:31:09.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:31:09.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:31:09.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:31:09.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:31:09.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:31:09.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:09.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:09.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:09.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:09.554 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:09.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:31:09.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:31:09.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:31:09.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:31:09.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:09.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:09.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:09.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:09.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:31:09.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:31:09.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:31:09.556 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:31:09.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dd8fdc29-fe74-48b8-a971-d570a6318a2f 2026-02-25T22:31:09.556 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 dd8fdc29-fe74-48b8-a971-d570a6318a2f 2026-02-25T22:31:09.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dd8fdc29-fe74-48b8-a971-d570a6318a2f' 2026-02-25T22:31:09.557 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:31:09.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAteJ9p+oOjIRAAxTXWLwHb7r41LG6JNoKbVg== 2026-02-25T22:31:09.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAteJ9p+oOjIRAAxTXWLwHb7r41LG6JNoKbVg=="}' 2026-02-25T22:31:09.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dd8fdc29-fe74-48b8-a971-d570a6318a2f -i td/test-erasure-eio/0/new.json 2026-02-25T22:31:09.680 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:31:09.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:31:09.689 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAteJ9p+oOjIRAAxTXWLwHb7r41LG6JNoKbVg== --osd-uuid dd8fdc29-fe74-48b8-a971-d570a6318a2f 2026-02-25T22:31:09.705 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:09.701+0000 7f05c5257a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:09.706 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:09.701+0000 7f05c5257a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:09.707 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:09.701+0000 7f05c5257a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:09.708 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:09.705+0000 7f05c5257a40 -1 bdev(0x55efbec82400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:31:09.708 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:09.705+0000 7f05c5257a40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:31:10.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:31:10.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:31:10.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:31:10.412 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:31:10.413 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:31:10.560 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:31:10.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:31:10.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:10.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:31:10.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:31:10.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:31:10.576 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:10.573+0000 7f63d54d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:10.577 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:10.573+0000 7f63d54d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:10.578 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:10.573+0000 7f63d54d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:10.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:31:10.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:10.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:31:10.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:10.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:10.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:10.669 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:31:10.669 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:10.669 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:10.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:10.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:10.969 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:10.965+0000 7f63d54d3a40 -1 Falling back to public interface 2026-02-25T22:31:11.529 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:11.525+0000 7f63d54d3a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:31:11.876 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:11.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:11.877 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:31:11.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:11.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:11.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:12.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:13.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:13.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:13.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:31:13.088 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:31:13.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:13.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:13.293 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2306861964,v1:127.0.0.1:6803/2306861964] [v2:127.0.0.1:6804/2306861964,v1:127.0.0.1:6805/2306861964] exists,up dd8fdc29-fe74-48b8-a971-d570a6318a2f 2026-02-25T22:31:13.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:13.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:13.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:13.298 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:31:13.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:31:13.300 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:31:13.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=43eeb66e-9505-4977-a033-051db0dedc3e 2026-02-25T22:31:13.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 43eeb66e-9505-4977-a033-051db0dedc3e' 2026-02-25T22:31:13.301 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 43eeb66e-9505-4977-a033-051db0dedc3e 2026-02-25T22:31:13.301 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:31:13.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAxeJ9pnsVsEhAAbtD9cxgOmJGirMgh0PV1sQ== 2026-02-25T22:31:13.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAxeJ9pnsVsEhAAbtD9cxgOmJGirMgh0PV1sQ=="}' 2026-02-25T22:31:13.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 43eeb66e-9505-4977-a033-051db0dedc3e -i td/test-erasure-eio/1/new.json 2026-02-25T22:31:13.523 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:31:13.533 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:31:13.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxeJ9pnsVsEhAAbtD9cxgOmJGirMgh0PV1sQ== --osd-uuid 43eeb66e-9505-4977-a033-051db0dedc3e 2026-02-25T22:31:13.550 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:13.545+0000 7f03cd1cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:13.551 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:13.545+0000 7f03cd1cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:13.552 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:13.549+0000 7f03cd1cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:13.553 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:13.549+0000 7f03cd1cba40 -1 bdev(0x5635a033c400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:31:13.553 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:13.549+0000 7f03cd1cba40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:31:14.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:31:14.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:31:14.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:31:14.185 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:31:14.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:31:14.477 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:31:14.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:31:14.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:14.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:31:14.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:31:14.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:31:14.500 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:14.496+0000 7f384dd39a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:14.502 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:14.496+0000 7f384dd39a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:14.503 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:14.496+0000 7f384dd39a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:14.687 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:31:14.687 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:14.687 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:31:14.687 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:14.687 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:14.687 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:14.687 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:14.688 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:31:14.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:14.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:31:14.885 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:14.880+0000 7f384dd39a40 -1 Falling back to public interface 2026-02-25T22:31:14.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:15.574 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:15.568+0000 7f384dd39a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:31:15.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:15.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:15.900 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:31:15.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:15.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:15.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:31:16.113 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:17.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:17.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:17.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:31:17.115 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:31:17.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:17.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:31:17.326 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1118742438,v1:127.0.0.1:6811/1118742438] [v2:127.0.0.1:6812/1118742438,v1:127.0.0.1:6813/1118742438] exists,up 43eeb66e-9505-4977-a033-051db0dedc3e 2026-02-25T22:31:17.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:17.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:17.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:17.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:31:17.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:17.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:31:17.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:31:17.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:31:17.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:31:17.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:17.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:17.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:17.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:17.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:31:17.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:31:17.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:31:17.333 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:31:17.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e58e69bb-4fb0-4138-a311-0a9224ff4eb6 2026-02-25T22:31:17.333 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 e58e69bb-4fb0-4138-a311-0a9224ff4eb6' 2026-02-25T22:31:17.333 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 e58e69bb-4fb0-4138-a311-0a9224ff4eb6 2026-02-25T22:31:17.334 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:31:17.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1eJ9pspZ8FBAAb5Ot7J6c+x+hhs1AAF+Wvw== 2026-02-25T22:31:17.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1eJ9pspZ8FBAAb5Ot7J6c+x+hhs1AAF+Wvw=="}' 2026-02-25T22:31:17.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e58e69bb-4fb0-4138-a311-0a9224ff4eb6 -i td/test-erasure-eio/2/new.json 2026-02-25T22:31:17.560 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:31:17.571 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:31:17.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1eJ9pspZ8FBAAb5Ot7J6c+x+hhs1AAF+Wvw== --osd-uuid e58e69bb-4fb0-4138-a311-0a9224ff4eb6 2026-02-25T22:31:17.596 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:17.592+0000 7f56ebfdea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:17.598 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:17.592+0000 7f56ebfdea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:17.599 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:17.596+0000 7f56ebfdea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:17.600 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:17.596+0000 7f56ebfdea40 -1 bdev(0x55be507fe400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:31:17.600 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:17.596+0000 7f56ebfdea40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:31:18.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:31:18.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:31:18.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:31:18.216 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:31:18.217 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:31:18.505 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:31:18.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:31:18.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:18.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:31:18.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:31:18.507 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:31:18.529 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:18.524+0000 7fd0b5f1ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:18.530 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:18.524+0000 7fd0b5f1ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:18.531 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:18.528+0000 7fd0b5f1ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:18.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:31:18.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:18.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:31:18.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:18.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:18.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:18.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:18.713 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:31:18.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:18.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:31:18.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:18.989 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:18.984+0000 7fd0b5f1ca40 -1 Falling back to public interface 2026-02-25T22:31:19.598 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:19.592+0000 7fd0b5f1ca40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:31:19.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:19.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:19.933 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:31:19.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:19.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:19.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:31:20.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:21.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:21.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:21.160 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:31:21.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:31:21.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:21.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:31:21.368 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/883676844,v1:127.0.0.1:6819/883676844] [v2:127.0.0.1:6820/883676844,v1:127.0.0.1:6821/883676844] exists,up e58e69bb-4fb0-4138-a311-0a9224ff4eb6 2026-02-25T22:31:21.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:21.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:21.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:21.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:31:21.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:31:21.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:31:21.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:31:21.371 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:31:21.372 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:31:21.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:21.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:21.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:21.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:21.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:31:21.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:31:21.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:31:21.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:31:21.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5e8dcdf0-91e3-4749-a638-0998eedea1cb 2026-02-25T22:31:21.375 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 5e8dcdf0-91e3-4749-a638-0998eedea1cb 2026-02-25T22:31:21.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 5e8dcdf0-91e3-4749-a638-0998eedea1cb' 2026-02-25T22:31:21.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:31:21.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA5eJ9pi2XfFhAAAL+bwGJ0nppsp3jzAAERBw== 2026-02-25T22:31:21.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA5eJ9pi2XfFhAAAL+bwGJ0nppsp3jzAAERBw=="}' 2026-02-25T22:31:21.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5e8dcdf0-91e3-4749-a638-0998eedea1cb -i td/test-erasure-eio/3/new.json 2026-02-25T22:31:21.598 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:31:21.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:31:21.609 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5eJ9pi2XfFhAAAL+bwGJ0nppsp3jzAAERBw== --osd-uuid 5e8dcdf0-91e3-4749-a638-0998eedea1cb 2026-02-25T22:31:21.625 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:21.620+0000 7f0b0ce62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:21.626 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:21.620+0000 7f0b0ce62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:21.627 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:21.624+0000 7f0b0ce62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:21.628 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:21.624+0000 7f0b0ce62a40 -1 bdev(0x558f6e3f8400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:31:21.628 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:21.624+0000 7f0b0ce62a40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:31:22.242 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:31:22.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:31:22.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:31:22.244 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:31:22.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:31:22.538 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:31:22.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:31:22.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:22.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:31:22.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:31:22.539 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:31:22.562 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:22.556+0000 7f246acefa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:22.564 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:22.560+0000 7f246acefa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:22.565 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:22.560+0000 7f246acefa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:22.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:31:22.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:22.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:31:22.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:22.750 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:22.750 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:22.750 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:22.750 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:31:22.750 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:22.751 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:31:22.941 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:22.936+0000 7f246acefa40 -1 Falling back to public interface 2026-02-25T22:31:22.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:23.614 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:23.608+0000 7f246acefa40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:31:23.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:23.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:23.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:23.963 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:31:23.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:23.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:31:24.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:25.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:25.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:25.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:31:25.178 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:31:25.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:25.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:31:25.386 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 24 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3320175261,v1:127.0.0.1:6827/3320175261] [v2:127.0.0.1:6828/3320175261,v1:127.0.0.1:6829/3320175261] exists,up 5e8dcdf0-91e3-4749-a638-0998eedea1cb 2026-02-25T22:31:25.387 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:25.387 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:25.387 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:25.387 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:31:25.387 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:31:25.387 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:31:25.388 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:25.388 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:25.388 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:25.388 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:31:25.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:31:25.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:31:25.443 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:31:25.449 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:31:25.451 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:31:10.969+0000 7f63d54d3a40 0 load: jerasure load: lrc 2026-02-25T22:31:25.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:325: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2026-02-25T22:31:25.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:326: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T22:31:25.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:31:25.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:31:25.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T22:31:25.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:31:25.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T22:31:25.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:31:25.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T22:31:25.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:31:25.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:31:26.077 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:31:26.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:31:27.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:31:27.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:31:27.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:31:27.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:31:27.089 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:31:27.089 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:27.089 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:27.090 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:27.090 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:27.090 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:27.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:31:27.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:31:27.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:31:27.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:31:27.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:27.199 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:27.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:31:27.406 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:27.406 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:31:27.406 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:31:27.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:27.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:27.407 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:27.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803781 2026-02-25T22:31:27.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803781 2026-02-25T22:31:27.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781' 2026-02-25T22:31:27.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:27.485 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:31:27.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607557 2026-02-25T22:31:27.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607557 2026-02-25T22:31:27.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607557' 2026-02-25T22:31:27.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:27.567 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:31:27.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444036 2026-02-25T22:31:27.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444036 2026-02-25T22:31:27.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607557 2-73014444036' 2026-02-25T22:31:27.646 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:27.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:31:27.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215107 2026-02-25T22:31:27.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215107 2026-02-25T22:31:27.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607557 2-73014444036 3-103079215107' 2026-02-25T22:31:27.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:27.726 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803781 2026-02-25T22:31:27.726 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:27.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:27.728 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803781 2026-02-25T22:31:27.728 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:27.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803781 2026-02-25T22:31:27.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803781' 2026-02-25T22:31:27.730 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803781 2026-02-25T22:31:27.730 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:27.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803779 -lt 25769803781 2026-02-25T22:31:27.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:28.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:31:28.949 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:29.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803781 2026-02-25T22:31:29.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:29.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607557 2026-02-25T22:31:29.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:29.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:31:29.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607557 2026-02-25T22:31:29.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:29.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607557 2026-02-25T22:31:29.166 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 51539607557 2026-02-25T22:31:29.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607557' 2026-02-25T22:31:29.167 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:31:29.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607557 -lt 51539607557 2026-02-25T22:31:29.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:29.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444036 2026-02-25T22:31:29.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:29.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:31:29.393 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444036 2026-02-25T22:31:29.393 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:29.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444036 2026-02-25T22:31:29.395 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 73014444036 2026-02-25T22:31:29.395 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444036' 2026-02-25T22:31:29.395 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:31:29.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444036 -lt 73014444036 2026-02-25T22:31:29.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:29.611 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-103079215107 2026-02-25T22:31:29.611 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:29.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:31:29.613 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-103079215107 2026-02-25T22:31:29.613 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:29.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=103079215107 2026-02-25T22:31:29.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 103079215107' 2026-02-25T22:31:29.615 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 103079215107 2026-02-25T22:31:29.615 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:31:29.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 103079215107 -lt 103079215107 2026-02-25T22:31:29.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:31:29.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:29.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:30.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:31:30.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:31:30.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:31:30.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:31:30.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:31:30.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:31:30.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:31:30.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:31:30.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:31:30.327 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:31:30.327 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:30.327 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:30.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:31:30.606 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:328: TEST_rados_get_bad_size_shard_0: local shard_id=0 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:329: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=10 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T22:31:30.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-73931-0-10 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-73931-0-10 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-73931-0-10 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:31:30.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-73931-0-10 td/test-erasure-eio/ORIGINAL 2026-02-25T22:31:30.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-73931-0-10 td/test-erasure-eio 0 10 set 2026-02-25T22:31:30.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-0-10 2026-02-25T22:31:30.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:31:30.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:31:30.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:31:30.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2026-02-25T22:31:30.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:31:30.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2026-02-25T22:31:30.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:31:30.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T22:31:30.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:31:30.626 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-0-10 2026-02-25T22:31:30.626 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:31:30.626 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-0-10 2026-02-25T22:31:30.627 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-0-10 2026-02-25T22:31:30.627 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:31:30.838 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:31:30.838 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:30.838 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:31:30.838 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:31:30.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:31:30.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:31:30.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2026-02-25T22:31:30.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:31:31.130 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:31:31.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T22:31:31.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2026-02-25T22:31:31.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-25T22:31:31.142 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records in 2026-02-25T22:31:31.142 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records out 2026-02-25T22:31:31.142 INFO:tasks.workunit.client.0.trial125.stderr:10 bytes copied, 3.4544e-05 s, 289 kB/s 2026-02-25T22:31:31.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-73931-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:31.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:31:31.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-73931-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:31.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:31.144 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:31.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:31.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-73931-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:31.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:31:31.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:31:31.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-25T22:31:31.451 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:31:31.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-25T22:31:31.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-73931-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:31.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-25T22:31:31.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:31:31.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:31:31.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T22:31:31.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:31:31.738 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:31.739 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:31.739 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:31.739 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:31.739 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:31.739 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:31.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:31.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:31.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:31:31.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:31:31.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:31.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:31.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:31.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:31.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:31:31.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:31:31.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:31:31.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T22:31:31.742 INFO:tasks.workunit.client.0.trial125.stderr:start osd.3 2026-02-25T22:31:31.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:31.742 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-25T22:31:31.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T22:31:31.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:31:31.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:31:31.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:31:31.758 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:31.752+0000 7fb1b03a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:31.759 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:31.756+0000 7fb1b03a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:31.760 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:31.756+0000 7fb1b03a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:31.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T22:31:31.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:31.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:31:31.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:31.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:31.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:31.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:31.953 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:31:31.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:31.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:31:32.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:32.185 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:32.180+0000 7fb1b03a5a40 -1 Falling back to public interface 2026-02-25T22:31:32.840 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:32.836+0000 7fb1b03a5a40 -1 osd.3 33 log_to_monitors true 2026-02-25T22:31:33.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:33.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:33.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:33.168 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:33.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:33.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:31:33.381 INFO:tasks.workunit.client.0.trial125.stderr:osd.3 up in weight 1 up_from 37 up_thru 37 down_at 34 last_clean_interval [24,33) [v2:127.0.0.1:6826/3932104902,v1:127.0.0.1:6827/3932104902] [v2:127.0.0.1:6828/3932104902,v1:127.0.0.1:6829/3932104902] exists,up 5e8dcdf0-91e3-4749-a638-0998eedea1cb 2026-02-25T22:31:33.382 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:33.382 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:33.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:33.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:31:33.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:31:33.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:31:33.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:31:33.383 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:31:33.384 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:33.384 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:33.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:33.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:33.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:33.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:31:33.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:31:33.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:31:33.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:31:33.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:33.490 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:33.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:31:33.703 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:33.703 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:31:33.703 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:31:33.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:33.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:33.704 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:33.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803785 2026-02-25T22:31:33.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803785 2026-02-25T22:31:33.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785' 2026-02-25T22:31:33.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:33.781 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:31:33.860 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607560 2026-02-25T22:31:33.860 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607560 2026-02-25T22:31:33.860 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-51539607560' 2026-02-25T22:31:33.860 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:33.861 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:31:33.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444039 2026-02-25T22:31:33.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444039 2026-02-25T22:31:33.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-51539607560 2-73014444039' 2026-02-25T22:31:33.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:33.940 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:31:34.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789955 2026-02-25T22:31:34.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789955 2026-02-25T22:31:34.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-51539607560 2-73014444039 3-158913789955' 2026-02-25T22:31:34.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:34.027 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803785 2026-02-25T22:31:34.027 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:34.027 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:34.028 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803785 2026-02-25T22:31:34.028 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:34.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803785 2026-02-25T22:31:34.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2026-02-25T22:31:34.028 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803785 2026-02-25T22:31:34.028 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:34.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803785 2026-02-25T22:31:34.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:35.246 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:31:35.246 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:35.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803785 2026-02-25T22:31:35.462 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:35.463 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607560 2026-02-25T22:31:35.463 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:35.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:31:35.464 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607560 2026-02-25T22:31:35.465 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:35.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607560 2026-02-25T22:31:35.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607560' 2026-02-25T22:31:35.466 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 51539607560 2026-02-25T22:31:35.467 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:31:35.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607560 -lt 51539607560 2026-02-25T22:31:35.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:35.693 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444039 2026-02-25T22:31:35.693 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:35.694 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:31:35.695 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444039 2026-02-25T22:31:35.695 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:35.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444039 2026-02-25T22:31:35.697 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444039' 2026-02-25T22:31:35.697 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 73014444039 2026-02-25T22:31:35.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:31:35.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444039 -lt 73014444039 2026-02-25T22:31:35.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:35.913 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-158913789955 2026-02-25T22:31:35.913 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:35.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:31:35.915 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-158913789955 2026-02-25T22:31:35.915 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:35.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=158913789955 2026-02-25T22:31:35.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 158913789955' 2026-02-25T22:31:35.916 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 158913789955 2026-02-25T22:31:35.917 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:31:36.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 158913789955 -lt 158913789955 2026-02-25T22:31:36.129 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:31:36.129 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:36.129 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:36.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:31:36.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:31:36.410 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:31:36.410 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:31:36.410 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:31:36.410 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:31:36.410 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:31:36.410 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:31:36.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:31:36.615 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:31:36.615 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:36.615 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:36.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:31:36.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:31:36.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:31:36.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:31:36.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:31:37.215 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:31:37.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-0-10 2026-02-25T22:31:37.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:31:37.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:31:37.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-0-10 2026-02-25T22:31:37.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:31:37.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:31:37.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-73931-0-10 td/test-erasure-eio/COPY 2026-02-25T22:31:37.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:31:37.255 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:31:37.256 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2026-02-25T22:31:37.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2026-02-25T22:31:37.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-73931-0-10 td/test-erasure-eio 1 10 set 2026-02-25T22:31:37.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-0-10 2026-02-25T22:31:37.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:31:37.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:31:37.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:31:37.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T22:31:37.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:31:37.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2026-02-25T22:31:37.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:31:37.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T22:31:37.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:31:37.260 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-0-10 2026-02-25T22:31:37.260 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:31:37.260 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-0-10 2026-02-25T22:31:37.261 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-0-10 2026-02-25T22:31:37.261 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:31:37.468 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:31:37.468 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:37.468 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:31:37.468 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:31:37.468 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:31:37.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:31:37.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T22:31:37.469 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:31:37.778 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:31:37.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T22:31:37.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2026-02-25T22:31:37.789 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-25T22:31:37.791 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records in 2026-02-25T22:31:37.791 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records out 2026-02-25T22:31:37.791 INFO:tasks.workunit.client.0.trial125.stderr:10 bytes copied, 3.206e-05 s, 312 kB/s 2026-02-25T22:31:37.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-73931-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:37.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:31:37.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-73931-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:37.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:38.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:38.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-73931-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:38.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:31:38.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:31:38.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T22:31:38.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:31:38.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T22:31:38.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-73931-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:38.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T22:31:38.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:31:38.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:31:38.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T22:31:38.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:31:38.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:31:38.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:31:38.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:38.406 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:38.407 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:38.408 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:38.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:38.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:38.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:31:38.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:31:38.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:38.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:38.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:38.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:38.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:31:38.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:31:38.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:31:38.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T22:31:38.409 INFO:tasks.workunit.client.0.trial125.stderr:start osd.1 2026-02-25T22:31:38.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:38.409 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T22:31:38.409 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T22:31:38.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:31:38.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:31:38.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:31:38.424 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:38.420+0000 7f134d338a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:38.426 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:38.420+0000 7f134d338a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:38.427 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:38.424+0000 7f134d338a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:38.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T22:31:38.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:38.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:31:38.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:38.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:38.621 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:38.622 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:38.622 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:31:38.622 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:38.622 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:31:38.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:38.869 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:38.864+0000 7f134d338a40 -1 Falling back to public interface 2026-02-25T22:31:39.553 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:39.548+0000 7f134d338a40 -1 osd.1 42 log_to_monitors true 2026-02-25T22:31:39.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:39.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:39.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:39.838 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:39.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:39.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:31:40.047 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 45 last_clean_interval [12,42) [v2:127.0.0.1:6810/1082208915,v1:127.0.0.1:6811/1082208915] [v2:127.0.0.1:6812/1082208915,v1:127.0.0.1:6813/1082208915] exists,up 43eeb66e-9505-4977-a033-051db0dedc3e 2026-02-25T22:31:40.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:40.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:40.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:40.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:31:40.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:31:40.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:31:40.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:31:40.048 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:31:40.049 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:40.049 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:40.049 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:40.049 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:40.050 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:40.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:31:40.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:31:40.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:31:40.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:31:40.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:40.146 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:40.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:31:40.358 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:40.359 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:31:40.359 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:31:40.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:40.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:40.359 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:40.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803788 2026-02-25T22:31:40.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803788 2026-02-25T22:31:40.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788' 2026-02-25T22:31:40.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:40.436 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:31:40.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=206158430211 2026-02-25T22:31:40.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 206158430211 2026-02-25T22:31:40.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-206158430211' 2026-02-25T22:31:40.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:40.516 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:31:40.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444042 2026-02-25T22:31:40.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444042 2026-02-25T22:31:40.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-206158430211 2-73014444042' 2026-02-25T22:31:40.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:40.597 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:31:40.676 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789958 2026-02-25T22:31:40.676 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789958 2026-02-25T22:31:40.677 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-206158430211 2-73014444042 3-158913789958' 2026-02-25T22:31:40.677 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:40.677 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803788 2026-02-25T22:31:40.677 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:40.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:40.679 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803788 2026-02-25T22:31:40.679 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:40.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803788 2026-02-25T22:31:40.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803788' 2026-02-25T22:31:40.681 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803788 2026-02-25T22:31:40.681 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:40.902 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803788 -lt 25769803788 2026-02-25T22:31:40.902 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:40.903 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-206158430211 2026-02-25T22:31:40.903 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:40.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:31:40.905 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-206158430211 2026-02-25T22:31:40.905 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:40.907 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=206158430211 2026-02-25T22:31:40.907 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 206158430211' 2026-02-25T22:31:40.907 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 206158430211 2026-02-25T22:31:40.907 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:31:41.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 206158430211 -lt 206158430211 2026-02-25T22:31:41.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:41.123 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444042 2026-02-25T22:31:41.123 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:41.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:31:41.125 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444042 2026-02-25T22:31:41.126 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:41.127 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444042 2026-02-25T22:31:41.127 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444042' 2026-02-25T22:31:41.127 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 73014444042 2026-02-25T22:31:41.127 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:31:41.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444041 -lt 73014444042 2026-02-25T22:31:41.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:42.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:31:42.349 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:31:42.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444041 -lt 73014444042 2026-02-25T22:31:42.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:43.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:31:43.563 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:31:43.776 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444043 -lt 73014444042 2026-02-25T22:31:43.776 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:43.777 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-158913789958 2026-02-25T22:31:43.777 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:43.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:31:43.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-158913789958 2026-02-25T22:31:43.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:43.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=158913789958 2026-02-25T22:31:43.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 158913789958' 2026-02-25T22:31:43.781 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 158913789958 2026-02-25T22:31:43.781 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:31:44.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 158913789958 -lt 158913789958 2026-02-25T22:31:44.009 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:31:44.009 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:44.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:44.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:31:44.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:31:44.282 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:31:44.283 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:31:44.283 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:31:44.283 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:31:44.283 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:31:44.283 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:31:44.483 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:31:44.483 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:31:44.484 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:44.484 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:44.757 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:31:44.757 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:31:44.757 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:31:44.757 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:31:44.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:31:45.039 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:31:45.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2026-02-25T22:31:45.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-0-10 fail 2026-02-25T22:31:45.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:31:45.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:31:45.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-0-10 2026-02-25T22:31:45.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-25T22:31:45.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-25T22:31:45.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-73931-0-10 td/test-erasure-eio/COPY 2026-02-25T22:31:45.076 INFO:tasks.workunit.client.0.trial125.stderr:error getting pool-jerasure/obj-size-73931-0-10: (5) Input/output error 2026-02-25T22:31:45.078 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-25T22:31:45.078 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T22:31:45.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:330: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2026-02-25T22:31:45.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T22:31:45.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=0 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-73931-0-0 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-73931-0-0 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-73931-0-0 2026-02-25T22:31:45.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:45.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:31:45.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:45.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:31:45.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:45.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:31:45.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:45.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:31:45.083 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-73931-0-0 td/test-erasure-eio/ORIGINAL 2026-02-25T22:31:45.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-73931-0-0 td/test-erasure-eio 0 0 set 2026-02-25T22:31:45.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-0-0 2026-02-25T22:31:45.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:31:45.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:31:45.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:31:45.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2026-02-25T22:31:45.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:31:45.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2026-02-25T22:31:45.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:31:45.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T22:31:45.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:31:45.100 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-0-0 2026-02-25T22:31:45.100 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:31:45.100 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-0-0 2026-02-25T22:31:45.100 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-0-0 2026-02-25T22:31:45.101 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:31:45.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:31:45.313 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:45.313 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:31:45.313 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:31:45.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:31:45.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:31:45.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2026-02-25T22:31:45.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:31:45.604 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:31:45.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T22:31:45.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2026-02-25T22:31:45.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-25T22:31:45.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-73931-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:45.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:31:45.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:31:45.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-25T22:31:45.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:31:45.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-73931-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:45.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:31:45.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:31:45.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-25T22:31:45.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:31:45.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-25T22:31:45.618 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:45.618 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:45.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:45.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:45.618 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:45.926 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:45.926 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-73931-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:45.926 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:31:45.926 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:31:45.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-25T22:31:45.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:31:45.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-25T22:31:45.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-73931-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:46.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-25T22:31:46.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:31:46.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:31:46.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T22:31:46.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:31:46.170 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:46.171 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:46.171 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:46.171 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:46.171 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:46.171 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:46.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:46.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:46.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:31:46.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:31:46.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:46.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:46.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:46.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:46.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:31:46.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:31:46.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:31:46.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T22:31:46.174 INFO:tasks.workunit.client.0.trial125.stderr:start osd.3 2026-02-25T22:31:46.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:46.174 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-25T22:31:46.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T22:31:46.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:31:46.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:31:46.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:31:46.189 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:46.184+0000 7f7b7270ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:46.190 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:46.184+0000 7f7b7270ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:46.191 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:46.184+0000 7f7b7270ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:46.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T22:31:46.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:46.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:31:46.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:46.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:46.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:46.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:46.378 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:31:46.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:46.378 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:31:46.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:46.645 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:46.640+0000 7f7b7270ca40 -1 Falling back to public interface 2026-02-25T22:31:47.206 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:47.200+0000 7f7b7270ca40 -1 osd.3 54 log_to_monitors true 2026-02-25T22:31:47.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:47.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:47.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:47.595 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:47.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:47.596 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:31:47.808 INFO:tasks.workunit.client.0.trial125.stderr:osd.3 up in weight 1 up_from 58 up_thru 58 down_at 55 last_clean_interval [37,54) [v2:127.0.0.1:6826/4267460301,v1:127.0.0.1:6827/4267460301] [v2:127.0.0.1:6828/4267460301,v1:127.0.0.1:6829/4267460301] exists,up 5e8dcdf0-91e3-4749-a638-0998eedea1cb 2026-02-25T22:31:47.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:47.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:47.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:47.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:31:47.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:31:47.809 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:31:47.810 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:31:47.810 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:31:47.810 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:47.810 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:47.811 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:47.811 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:47.811 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:47.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:31:47.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:31:47.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:31:47.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:31:47.916 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:47.916 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:48.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:31:48.120 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:48.120 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:31:48.121 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:31:48.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:48.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:48.121 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:48.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803792 2026-02-25T22:31:48.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803792 2026-02-25T22:31:48.199 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803792' 2026-02-25T22:31:48.200 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:48.200 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:31:48.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=206158430214 2026-02-25T22:31:48.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 206158430214 2026-02-25T22:31:48.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803792 1-206158430214' 2026-02-25T22:31:48.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:48.280 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:31:48.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444046 2026-02-25T22:31:48.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444046 2026-02-25T22:31:48.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803792 1-206158430214 2-73014444046' 2026-02-25T22:31:48.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:48.358 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:31:48.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103171 2026-02-25T22:31:48.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103171 2026-02-25T22:31:48.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803792 1-206158430214 2-73014444046 3-249108103171' 2026-02-25T22:31:48.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:48.439 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803792 2026-02-25T22:31:48.439 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:48.440 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:48.441 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803792 2026-02-25T22:31:48.441 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:48.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803792 2026-02-25T22:31:48.443 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803792' 2026-02-25T22:31:48.443 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803792 2026-02-25T22:31:48.443 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:48.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803792 -lt 25769803792 2026-02-25T22:31:48.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:48.663 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-206158430214 2026-02-25T22:31:48.663 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:48.664 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:31:48.665 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-206158430214 2026-02-25T22:31:48.665 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:48.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=206158430214 2026-02-25T22:31:48.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 206158430214' 2026-02-25T22:31:48.667 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 206158430214 2026-02-25T22:31:48.667 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:31:48.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 206158430214 -lt 206158430214 2026-02-25T22:31:48.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:48.887 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444046 2026-02-25T22:31:48.887 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:48.888 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:31:48.889 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444046 2026-02-25T22:31:48.889 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:48.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444046 2026-02-25T22:31:48.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444046' 2026-02-25T22:31:48.891 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 73014444046 2026-02-25T22:31:48.891 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:31:49.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444046 -lt 73014444046 2026-02-25T22:31:49.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:49.111 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-249108103171 2026-02-25T22:31:49.112 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:49.113 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:31:49.114 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-249108103171 2026-02-25T22:31:49.114 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:49.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=249108103171 2026-02-25T22:31:49.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 249108103171' 2026-02-25T22:31:49.115 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 249108103171 2026-02-25T22:31:49.116 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:31:49.328 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 249108103171 -lt 249108103171 2026-02-25T22:31:49.329 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:31:49.329 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:49.329 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:49.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:31:49.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:31:49.608 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:31:49.608 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:31:49.608 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:31:49.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:31:49.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:31:49.609 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:31:49.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:31:49.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:31:49.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:49.818 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:50.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:31:50.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:31:50.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:31:50.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:31:50.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:31:50.357 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:31:50.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-0-0 2026-02-25T22:31:50.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:31:50.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:31:50.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-0-0 2026-02-25T22:31:50.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:31:50.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:31:50.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-73931-0-0 td/test-erasure-eio/COPY 2026-02-25T22:31:50.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:31:50.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:31:50.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2026-02-25T22:31:50.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2026-02-25T22:31:50.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-73931-0-0 td/test-erasure-eio 1 0 set 2026-02-25T22:31:50.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-0-0 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-0-0 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:31:50.394 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-0-0 2026-02-25T22:31:50.395 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-0-0 2026-02-25T22:31:50.395 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:31:50.600 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:31:50.600 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:50.600 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:31:50.600 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:31:50.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:31:50.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:31:50.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T22:31:50.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:31:50.862 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:31:50.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T22:31:50.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2026-02-25T22:31:50.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-25T22:31:50.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-73931-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:50.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:31:50.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:31:50.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T22:31:50.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:31:50.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-73931-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:50.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:31:50.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:31:50.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T22:31:50.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:31:50.876 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T22:31:50.876 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:50.876 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:50.876 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:50.876 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:50.876 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:51.183 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:51.183 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-73931-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:51.183 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:31:51.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:31:51.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T22:31:51.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:31:51.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T22:31:51.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-73931-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:51.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T22:31:51.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:31:51.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:31:51.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T22:31:51.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:31:51.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:31:51.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:31:51.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:51.425 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:51.426 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:51.428 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:51.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:51.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:51.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:31:51.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:31:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:31:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:31:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:31:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T22:31:51.429 INFO:tasks.workunit.client.0.trial125.stderr:start osd.1 2026-02-25T22:31:51.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:51.429 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T22:31:51.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T22:31:51.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:31:51.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:31:51.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:31:51.446 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:51.440+0000 7f73e5e10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:51.448 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:51.444+0000 7f73e5e10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:51.449 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:51.444+0000 7f73e5e10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:51.640 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T22:31:51.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:51.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:31:51.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:51.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:51.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:51.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:51.641 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:31:51.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:51.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:31:51.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:51.881 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:51.876+0000 7f73e5e10a40 -1 Falling back to public interface 2026-02-25T22:31:52.418 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:52.412+0000 7f73e5e10a40 -1 osd.1 64 log_to_monitors true 2026-02-25T22:31:52.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:52.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:52.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:52.853 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:52.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:52.854 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:31:53.056 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [48,64) [v2:127.0.0.1:6810/2773435973,v1:127.0.0.1:6811/2773435973] [v2:127.0.0.1:6812/2773435973,v1:127.0.0.1:6813/2773435973] exists,up 43eeb66e-9505-4977-a033-051db0dedc3e 2026-02-25T22:31:53.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:53.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:53.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:53.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:31:53.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:31:53.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:31:53.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:31:53.058 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:31:53.059 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:53.059 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:53.059 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:53.059 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:53.059 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:53.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:31:53.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:31:53.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:31:53.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:31:53.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:53.169 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:53.387 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:31:53.387 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:53.387 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:31:53.387 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:31:53.387 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:53.388 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:53.388 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:53.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803795 2026-02-25T22:31:53.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803795 2026-02-25T22:31:53.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803795' 2026-02-25T22:31:53.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:53.465 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:31:53.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776131 2026-02-25T22:31:53.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776131 2026-02-25T22:31:53.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803795 1-292057776131' 2026-02-25T22:31:53.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:53.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:31:53.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444049 2026-02-25T22:31:53.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444049 2026-02-25T22:31:53.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803795 1-292057776131 2-73014444049' 2026-02-25T22:31:53.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:53.629 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:31:53.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103174 2026-02-25T22:31:53.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103174 2026-02-25T22:31:53.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803795 1-292057776131 2-73014444049 3-249108103174' 2026-02-25T22:31:53.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:53.706 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803795 2026-02-25T22:31:53.706 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:53.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:53.708 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803795 2026-02-25T22:31:53.708 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:53.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803795 2026-02-25T22:31:53.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803795' 2026-02-25T22:31:53.710 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803795 2026-02-25T22:31:53.710 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:53.924 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803792 -lt 25769803795 2026-02-25T22:31:53.924 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:54.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:31:54.926 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:55.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803795 -lt 25769803795 2026-02-25T22:31:55.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:55.139 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-292057776131 2026-02-25T22:31:55.139 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:55.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:31:55.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-292057776131 2026-02-25T22:31:55.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:55.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776131 2026-02-25T22:31:55.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 292057776131' 2026-02-25T22:31:55.143 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 292057776131 2026-02-25T22:31:55.143 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:31:55.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776131 -lt 292057776131 2026-02-25T22:31:55.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:55.353 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444049 2026-02-25T22:31:55.353 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:55.354 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:31:55.355 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444049 2026-02-25T22:31:55.355 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:55.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444049 2026-02-25T22:31:55.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444049' 2026-02-25T22:31:55.357 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 73014444049 2026-02-25T22:31:55.357 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:31:55.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444049 -lt 73014444049 2026-02-25T22:31:55.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:55.572 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-249108103174 2026-02-25T22:31:55.573 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:55.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:31:55.574 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-249108103174 2026-02-25T22:31:55.575 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:55.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=249108103174 2026-02-25T22:31:55.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 249108103174' 2026-02-25T22:31:55.576 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 249108103174 2026-02-25T22:31:55.577 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:31:55.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 249108103174 -lt 249108103174 2026-02-25T22:31:55.791 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:31:55.792 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:55.792 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:56.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:31:56.077 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:31:56.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:31:56.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:31:56.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:31:56.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:31:56.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:31:56.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:31:56.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:31:56.290 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:31:56.290 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:56.290 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:56.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:31:56.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:31:56.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:31:56.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:31:56.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:31:56.837 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:31:56.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2026-02-25T22:31:56.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-0-0 fail 2026-02-25T22:31:56.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:31:56.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:31:56.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-0-0 2026-02-25T22:31:56.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-25T22:31:56.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-25T22:31:56.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-73931-0-0 td/test-erasure-eio/COPY 2026-02-25T22:31:56.872 INFO:tasks.workunit.client.0.trial125.stderr:error getting pool-jerasure/obj-size-73931-0-0: (5) Input/output error 2026-02-25T22:31:56.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-25T22:31:56.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T22:31:56.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:331: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2026-02-25T22:31:56.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T22:31:56.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=256 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=add 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-73931-0-256 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-73931-0-256 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-73931-0-256 2026-02-25T22:31:56.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:56.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:31:56.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:56.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:31:56.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:56.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:31:56.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:31:56.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:31:56.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-73931-0-256 td/test-erasure-eio/ORIGINAL 2026-02-25T22:31:56.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-73931-0-256 td/test-erasure-eio 0 256 add 2026-02-25T22:31:56.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-0-256 2026-02-25T22:31:56.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:31:56.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:31:56.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:31:56.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2026-02-25T22:31:56.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:31:56.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2026-02-25T22:31:56.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:31:56.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2026-02-25T22:31:56.898 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:31:56.898 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-0-256 2026-02-25T22:31:56.898 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:31:56.898 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-0-256 2026-02-25T22:31:56.898 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-0-256 2026-02-25T22:31:56.899 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:31:57.106 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:31:57.106 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:57.106 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:31:57.107 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:31:57.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:31:57.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:31:57.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2026-02-25T22:31:57.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:31:57.399 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:31:57.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2026-02-25T22:31:57.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-73931-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:57.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:31:57.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:31:57.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-25T22:31:57.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:31:57.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-73931-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:57.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:31:57.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:31:57.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-25T22:31:57.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:31:57.413 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-25T22:31:57.413 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:57.413 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:57.413 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:57.413 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:57.413 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:57.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:57.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-73931-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:57.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:31:57.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:31:57.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-25T22:31:57.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:31:57.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-25T22:31:57.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-73931-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:31:57.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-25T22:31:57.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:31:57.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:31:57.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T22:31:57.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:31:57.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:31:57.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:31:57.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:57.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:57.878 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:31:57.879 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:57.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:31:57.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:57.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:31:57.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr:start osd.3 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-25T22:31:57.881 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T22:31:57.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:31:57.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:31:57.882 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:31:57.896 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:57.892+0000 7fa2835f1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:57.898 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:57.892+0000 7fa2835f1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:57.899 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:57.892+0000 7fa2835f1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:58.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T22:31:58.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:58.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:31:58.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:58.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:58.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:58.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:58.095 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:31:58.095 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:58.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:31:58.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:58.361 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:58.356+0000 7fa2835f1a40 -1 Falling back to public interface 2026-02-25T22:31:58.990 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:31:58.984+0000 7fa2835f1a40 -1 osd.3 71 log_to_monitors true 2026-02-25T22:31:59.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:59.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:59.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:59.307 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:59.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:59.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:31:59.515 INFO:tasks.workunit.client.0.trial125.stderr:osd.3 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [58,71) [v2:127.0.0.1:6826/4272498618,v1:127.0.0.1:6827/4272498618] [v2:127.0.0.1:6828/4272498618,v1:127.0.0.1:6829/4272498618] exists,up 5e8dcdf0-91e3-4749-a638-0998eedea1cb 2026-02-25T22:31:59.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:59.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:59.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:59.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:31:59.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:31:59.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:31:59.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:31:59.515 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:31:59.516 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:59.516 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:59.517 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:59.517 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:59.517 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:59.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:31:59.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:31:59.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:31:59.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:31:59.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:59.626 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:59.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:31:59.836 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:31:59.836 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:31:59.836 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:31:59.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:59.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:59.837 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:59.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803798 2026-02-25T22:31:59.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803798 2026-02-25T22:31:59.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803798' 2026-02-25T22:31:59.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:59.913 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:31:59.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776134 2026-02-25T22:31:59.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776134 2026-02-25T22:31:59.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803798 1-292057776134' 2026-02-25T22:31:59.989 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:59.989 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:32:00.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444052 2026-02-25T22:32:00.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444052 2026-02-25T22:32:00.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803798 1-292057776134 2-73014444052' 2026-02-25T22:32:00.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:00.069 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:32:00.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481795 2026-02-25T22:32:00.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481795 2026-02-25T22:32:00.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803798 1-292057776134 2-73014444052 3-330712481795' 2026-02-25T22:32:00.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:00.146 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803798 2026-02-25T22:32:00.146 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:00.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:32:00.147 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803798 2026-02-25T22:32:00.147 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:00.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803798 2026-02-25T22:32:00.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803798' 2026-02-25T22:32:00.149 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803798 2026-02-25T22:32:00.149 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:32:00.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803796 -lt 25769803798 2026-02-25T22:32:00.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:32:01.370 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:32:01.371 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:32:01.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803798 -lt 25769803798 2026-02-25T22:32:01.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:01.578 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-292057776134 2026-02-25T22:32:01.578 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:01.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:32:01.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-292057776134 2026-02-25T22:32:01.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:01.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776134 2026-02-25T22:32:01.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 292057776134' 2026-02-25T22:32:01.582 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 292057776134 2026-02-25T22:32:01.582 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:32:01.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776134 -lt 292057776134 2026-02-25T22:32:01.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:01.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444052 2026-02-25T22:32:01.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:01.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:32:01.799 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444052 2026-02-25T22:32:01.799 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:01.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444052 2026-02-25T22:32:01.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444052' 2026-02-25T22:32:01.800 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 73014444052 2026-02-25T22:32:01.801 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:32:02.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444052 -lt 73014444052 2026-02-25T22:32:02.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:02.009 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-330712481795 2026-02-25T22:32:02.009 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:02.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:32:02.011 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-330712481795 2026-02-25T22:32:02.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:02.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=330712481795 2026-02-25T22:32:02.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 330712481795' 2026-02-25T22:32:02.013 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 330712481795 2026-02-25T22:32:02.013 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:32:02.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 330712481795 -lt 330712481795 2026-02-25T22:32:02.232 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:32:02.232 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:02.233 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:02.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:32:02.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:02.516 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:02.516 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:02.516 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:02.516 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:02.516 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:02.516 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:02.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:32:02.724 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:02.725 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:02.725 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:03.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T22:32:03.004 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-25T22:32:03.004 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-25T22:32:03.004 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-25T22:32:03.004 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:32:03.004 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-25T22:32:03.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:32:03.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:03.105 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:03.105 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:03.105 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:03.105 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:03.105 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:03.105 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:03.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:32:03.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:03.310 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:03.311 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:03.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T22:32:03.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-25T22:32:03.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:32:03.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:32:03.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:32:03.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:32:03.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:32:03.586 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:32:03.586 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:32:03.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:32:03.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:32:03.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-25T22:32:03.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:32:03.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-25T22:32:04.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:32:04.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:04.065 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:04.065 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:04.065 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:04.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:04.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:04.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:04.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:32:04.271 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:04.272 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:04.272 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:04.551 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T22:32:04.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-25T22:32:04.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:32:04.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:32:04.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:32:04.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:32:04.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:32:04.553 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:32:04.553 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:32:04.833 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:32:04.833 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:32:04.833 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-25T22:32:04.833 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:32:04.833 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-25T22:32:05.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:32:05.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:05.235 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:05.235 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:05.235 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:05.235 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:05.236 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:05.236 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:05.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:32:05.447 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:05.447 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:05.447 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:05.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:32:05.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:32:05.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:32:05.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-25T22:32:05.723 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records in 2026-02-25T22:32:05.723 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records out 2026-02-25T22:32:05.723 INFO:tasks.workunit.client.0.trial125.stderr:256 bytes copied, 3.4785e-05 s, 7.4 MB/s 2026-02-25T22:32:05.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-73931-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:05.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:32:05.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:32:05.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-25T22:32:05.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:32:05.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-73931-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:05.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:32:05.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:32:05.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-25T22:32:05.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:32:05.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-25T22:32:05.724 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:32:05.725 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:32:05.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:32:05.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:32:05.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:32:06.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:32:06.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-73931-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:06.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:32:06.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:32:06.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-25T22:32:06.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:32:06.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-25T22:32:06.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-73931-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:06.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-25T22:32:06.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:32:06.303 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:32:06.304 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:32:06.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:32:06.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:32:06.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:32:06.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T22:32:06.306 INFO:tasks.workunit.client.0.trial125.stderr:start osd.3 2026-02-25T22:32:06.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:32:06.306 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-25T22:32:06.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T22:32:06.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:32:06.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:32:06.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:32:06.324 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:06.320+0000 7f25c97c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:06.325 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:06.320+0000 7f25c97c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:06.326 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:06.320+0000 7f25c97c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:06.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T22:32:06.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:32:06.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:32:06.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:32:06.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:32:06.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:06.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:32:06.514 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:32:06.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:06.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:32:06.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:32:06.785 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:06.780+0000 7f25c97c0a40 -1 Falling back to public interface 2026-02-25T22:32:07.374 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:07.368+0000 7f25c97c0a40 -1 osd.3 79 log_to_monitors true 2026-02-25T22:32:07.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:32:07.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:07.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:32:07.726 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:07.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:07.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:32:07.935 INFO:tasks.workunit.client.0.trial125.stderr:osd.3 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [77,79) [v2:127.0.0.1:6826/737654653,v1:127.0.0.1:6827/737654653] [v2:127.0.0.1:6828/737654653,v1:127.0.0.1:6829/737654653] exists,up 5e8dcdf0-91e3-4749-a638-0998eedea1cb 2026-02-25T22:32:07.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:32:07.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:32:07.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:32:07.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:32:07.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:32:07.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:32:07.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:32:07.937 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:32:07.938 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:32:07.938 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:32:07.938 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:32:07.938 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:32:07.938 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:32:08.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:32:08.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:32:08.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:32:08.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:32:08.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:32:08.050 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:32:08.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:32:08.263 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:08.263 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:32:08.263 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:32:08.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:32:08.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:08.263 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:32:08.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803802 2026-02-25T22:32:08.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803802 2026-02-25T22:32:08.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803802' 2026-02-25T22:32:08.342 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:08.342 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:32:08.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776138 2026-02-25T22:32:08.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776138 2026-02-25T22:32:08.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803802 1-292057776138' 2026-02-25T22:32:08.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:08.423 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:32:08.501 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444056 2026-02-25T22:32:08.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444056 2026-02-25T22:32:08.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803802 1-292057776138 2-73014444056' 2026-02-25T22:32:08.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:08.502 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:32:08.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285571 2026-02-25T22:32:08.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285571 2026-02-25T22:32:08.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803802 1-292057776138 2-73014444056 3-356482285571' 2026-02-25T22:32:08.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:08.581 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803802 2026-02-25T22:32:08.581 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:08.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:32:08.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803802 2026-02-25T22:32:08.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:08.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803802 2026-02-25T22:32:08.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803802' 2026-02-25T22:32:08.585 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803802 2026-02-25T22:32:08.585 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:32:08.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803802 -lt 25769803802 2026-02-25T22:32:08.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:08.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-292057776138 2026-02-25T22:32:08.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:08.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:32:08.799 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-292057776138 2026-02-25T22:32:08.799 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:08.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776138 2026-02-25T22:32:08.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 292057776138' 2026-02-25T22:32:08.801 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 292057776138 2026-02-25T22:32:08.801 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:32:09.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776138 -lt 292057776138 2026-02-25T22:32:09.024 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:09.025 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444056 2026-02-25T22:32:09.025 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:09.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:32:09.027 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444056 2026-02-25T22:32:09.027 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:09.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444056 2026-02-25T22:32:09.028 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444056' 2026-02-25T22:32:09.028 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 73014444056 2026-02-25T22:32:09.029 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:32:09.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444056 -lt 73014444056 2026-02-25T22:32:09.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:09.247 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-356482285571 2026-02-25T22:32:09.248 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:09.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:32:09.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-356482285571 2026-02-25T22:32:09.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:09.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=356482285571 2026-02-25T22:32:09.252 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 356482285571' 2026-02-25T22:32:09.252 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 356482285571 2026-02-25T22:32:09.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:32:09.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 356482285571 -lt 356482285571 2026-02-25T22:32:09.464 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:32:09.465 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:09.465 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:09.752 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:32:09.752 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:09.753 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:09.753 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:09.753 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:09.754 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:09.754 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:09.754 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:09.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:32:09.961 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:09.962 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:09.962 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:10.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T22:32:10.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-25T22:32:10.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-25T22:32:10.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-25T22:32:10.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:32:10.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-25T22:32:10.346 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:32:10.346 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:10.347 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:10.347 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:10.347 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:10.347 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:10.347 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:10.347 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:10.557 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:32:10.558 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:10.558 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:10.558 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:10.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T22:32:10.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-25T22:32:10.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:32:10.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:32:10.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:32:10.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:32:10.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:32:10.843 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:32:10.843 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:32:11.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:32:11.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:32:11.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-25T22:32:11.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:32:11.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-25T22:32:11.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:32:11.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:11.326 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:11.326 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:11.326 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:11.326 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:11.326 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:11.327 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:11.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:32:11.534 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:11.535 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:11.535 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:11.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T22:32:11.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-25T22:32:11.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:32:11.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:32:11.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:32:11.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:32:11.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:32:11.819 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:32:11.819 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:32:12.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:32:12.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:32:12.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-25T22:32:12.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:32:12.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-25T22:32:12.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:32:12.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:12.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:12.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:12.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:12.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:12.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:12.504 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:12.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:32:12.712 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:12.713 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:12.713 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:12.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T22:32:12.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-25T22:32:12.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:32:12.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:32:12.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:32:12.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:32:12.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:32:12.995 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:32:12.995 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:32:13.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:32:13.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:32:13.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-25T22:32:13.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:32:13.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-25T22:32:14.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:32:14.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:14.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:14.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:14.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:14.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:14.083 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:14.083 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:14.291 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:32:14.291 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:14.292 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:14.292 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:14.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T22:32:14.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-25T22:32:14.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:32:14.573 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:32:14.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:32:14.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:32:14.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:32:14.574 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:32:14.574 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:32:14.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:32:14.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:32:14.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 4 >= 13 )) 2026-02-25T22:32:14.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:32:14.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 1.6 2026-02-25T22:32:16.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:32:16.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:16.451 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:16.451 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:16.451 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:16.451 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:16.452 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:16.452 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:16.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:32:16.656 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:16.656 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:16.657 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:16.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:32:16.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:32:16.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:32:16.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:32:16.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:32:17.250 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:32:17.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-0-256 2026-02-25T22:32:17.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:32:17.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:32:17.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-0-256 2026-02-25T22:32:17.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:32:17.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:32:17.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-73931-0-256 td/test-erasure-eio/COPY 2026-02-25T22:32:17.290 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:32:17.291 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:32:17.293 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2026-02-25T22:32:17.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2026-02-25T22:32:17.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-73931-0-256 td/test-erasure-eio 1 256 add 2026-02-25T22:32:17.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-0-256 2026-02-25T22:32:17.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:32:17.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:32:17.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:32:17.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T22:32:17.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:32:17.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2026-02-25T22:32:17.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:32:17.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2026-02-25T22:32:17.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:32:17.296 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-0-256 2026-02-25T22:32:17.296 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:32:17.296 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-0-256 2026-02-25T22:32:17.296 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-0-256 2026-02-25T22:32:17.297 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:32:17.504 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:32:17.504 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:17.504 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:32:17.504 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:32:17.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:32:17.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:32:17.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T22:32:17.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:32:17.811 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:32:17.825 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2026-02-25T22:32:17.825 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-73931-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:17.825 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-73931-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:32:17.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:32:17.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:32:17.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:32:18.134 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:32:18.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-73931-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:18.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:32:18.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:32:18.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T22:32:18.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:32:18.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T22:32:18.135 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-73931-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:18.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T22:32:18.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:32:18.304 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:32:18.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T22:32:18.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:32:18.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:32:18.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:32:18.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:32:18.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:32:18.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:32:18.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:32:18.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:32:18.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:32:18.305 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:32:18.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:32:18.306 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:32:18.306 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:32:18.306 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:32:18.306 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:32:18.306 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:18.306 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:18.308 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:18.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:18.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:32:18.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:32:18.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:32:18.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:32:18.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:32:18.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:32:18.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:32:18.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:32:18.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:32:18.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:32:18.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T22:32:18.309 INFO:tasks.workunit.client.0.trial125.stderr:start osd.1 2026-02-25T22:32:18.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:32:18.309 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T22:32:18.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T22:32:18.309 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:32:18.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:32:18.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:32:18.324 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:18.319+0000 7fa344daba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:18.325 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:18.319+0000 7fa344daba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:18.326 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:18.319+0000 7fa344daba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:18.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T22:32:18.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:32:18.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:32:18.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:32:18.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:32:18.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:18.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:32:18.524 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:32:18.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:18.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:32:18.734 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:32:18.785 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:18.779+0000 7fa344daba40 -1 Falling back to public interface 2026-02-25T22:32:19.351 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:19.343+0000 7fa344daba40 -1 osd.1 86 log_to_monitors true 2026-02-25T22:32:19.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:32:19.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:19.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:32:19.736 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:19.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:19.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:32:19.942 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 up in weight 1 up_from 92 up_thru 92 down_at 89 last_clean_interval [68,86) [v2:127.0.0.1:6810/1952015759,v1:127.0.0.1:6811/1952015759] [v2:127.0.0.1:6812/1952015759,v1:127.0.0.1:6813/1952015759] exists,up 43eeb66e-9505-4977-a033-051db0dedc3e 2026-02-25T22:32:19.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:32:19.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:32:19.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:32:19.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:32:19.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:32:19.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:32:19.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:32:19.944 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:32:19.945 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:32:19.945 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:32:19.945 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:32:19.945 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:32:19.945 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:32:20.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:32:20.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:32:20.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:32:20.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:32:20.049 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:32:20.049 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:32:20.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:32:20.267 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:20.267 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:32:20.267 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:32:20.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:32:20.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:20.268 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:32:20.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803806 2026-02-25T22:32:20.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803806 2026-02-25T22:32:20.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803806' 2026-02-25T22:32:20.345 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:20.345 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:32:20.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991235 2026-02-25T22:32:20.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991235 2026-02-25T22:32:20.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803806 1-395136991235' 2026-02-25T22:32:20.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:20.422 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:32:20.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444060 2026-02-25T22:32:20.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444060 2026-02-25T22:32:20.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803806 1-395136991235 2-73014444060' 2026-02-25T22:32:20.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:20.501 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:32:20.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285575 2026-02-25T22:32:20.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285575 2026-02-25T22:32:20.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803806 1-395136991235 2-73014444060 3-356482285575' 2026-02-25T22:32:20.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:20.577 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803806 2026-02-25T22:32:20.577 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:20.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:32:20.579 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803806 2026-02-25T22:32:20.579 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:20.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803806 2026-02-25T22:32:20.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803806' 2026-02-25T22:32:20.581 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803806 2026-02-25T22:32:20.581 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:32:20.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803806 -lt 25769803806 2026-02-25T22:32:20.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:20.798 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-395136991235 2026-02-25T22:32:20.798 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:20.799 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:32:20.800 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-395136991235 2026-02-25T22:32:20.800 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:20.802 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=395136991235 2026-02-25T22:32:20.802 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 395136991235' 2026-02-25T22:32:20.802 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 395136991235 2026-02-25T22:32:20.802 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:32:21.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 395136991235 -lt 395136991235 2026-02-25T22:32:21.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:21.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444060 2026-02-25T22:32:21.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:21.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:32:21.022 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444060 2026-02-25T22:32:21.022 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:21.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444060 2026-02-25T22:32:21.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444060' 2026-02-25T22:32:21.024 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 73014444060 2026-02-25T22:32:21.024 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:32:21.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444060 -lt 73014444060 2026-02-25T22:32:21.231 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:21.232 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-356482285575 2026-02-25T22:32:21.232 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:21.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:32:21.234 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-356482285575 2026-02-25T22:32:21.234 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:21.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=356482285575 2026-02-25T22:32:21.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 356482285575' 2026-02-25T22:32:21.235 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 356482285575 2026-02-25T22:32:21.235 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:32:21.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 356482285575 -lt 356482285575 2026-02-25T22:32:21.453 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:32:21.453 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:21.453 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:21.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:32:21.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:21.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:21.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:21.728 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:21.728 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:21.728 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:21.728 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:21.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:32:21.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:21.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:21.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:22.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:32:22.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:32:22.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:32:22.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-25T22:32:22.213 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records in 2026-02-25T22:32:22.213 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records out 2026-02-25T22:32:22.213 INFO:tasks.workunit.client.0.trial125.stderr:256 bytes copied, 4.6096e-05 s, 5.6 MB/s 2026-02-25T22:32:22.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-73931-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:22.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-73931-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:32:22.214 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:32:22.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:32:22.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:32:22.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:32:22.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-73931-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:22.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:32:22.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:32:22.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T22:32:22.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:32:22.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T22:32:22.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-73931-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:22.813 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T22:32:22.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:32:22.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:32:22.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T22:32:22.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:32:22.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:32:22.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:32:22.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:32:22.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:32:22.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:32:22.814 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:32:22.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:32:22.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:32:22.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:32:22.815 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:32:22.815 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:32:22.815 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:32:22.815 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:32:22.815 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:32:22.815 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:22.815 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:22.815 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:32:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:32:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:32:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:32:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:32:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:32:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:32:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:32:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:32:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:32:22.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T22:32:22.819 INFO:tasks.workunit.client.0.trial125.stderr:start osd.1 2026-02-25T22:32:22.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:32:22.819 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T22:32:22.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T22:32:22.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:32:22.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:32:22.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:32:22.834 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:22.827+0000 7f2e14e2da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:22.836 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:22.831+0000 7f2e14e2da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:22.837 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:22.831+0000 7f2e14e2da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:23.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T22:32:23.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:32:23.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:32:23.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:32:23.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:32:23.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:23.033 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:32:23.033 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:32:23.033 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:23.033 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:32:23.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:32:23.289 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:23.283+0000 7f2e14e2da40 -1 Falling back to public interface 2026-02-25T22:32:23.958 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:23.951+0000 7f2e14e2da40 -1 osd.1 93 log_to_monitors true 2026-02-25T22:32:24.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:32:24.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:24.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:32:24.245 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:24.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:24.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:32:24.453 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 up in weight 1 up_from 97 up_thru 97 down_at 94 last_clean_interval [92,93) [v2:127.0.0.1:6810/3180155541,v1:127.0.0.1:6811/3180155541] [v2:127.0.0.1:6812/3180155541,v1:127.0.0.1:6813/3180155541] exists,up 43eeb66e-9505-4977-a033-051db0dedc3e 2026-02-25T22:32:24.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:32:24.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:32:24.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:32:24.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:32:24.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:32:24.454 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:32:24.454 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:32:24.454 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:32:24.455 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:32:24.455 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:32:24.455 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:32:24.455 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:32:24.455 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:32:24.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:32:24.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:32:24.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:32:24.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:32:24.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:32:24.565 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:32:24.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:32:24.774 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:24.775 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:32:24.775 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:32:24.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:32:24.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:24.775 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:32:24.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803809 2026-02-25T22:32:24.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803809 2026-02-25T22:32:24.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803809' 2026-02-25T22:32:24.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:24.857 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:32:24.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=416611827715 2026-02-25T22:32:24.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 416611827715 2026-02-25T22:32:24.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803809 1-416611827715' 2026-02-25T22:32:24.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:24.933 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:32:25.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444063 2026-02-25T22:32:25.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444063 2026-02-25T22:32:25.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803809 1-416611827715 2-73014444063' 2026-02-25T22:32:25.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:25.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:32:25.089 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285578 2026-02-25T22:32:25.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285578 2026-02-25T22:32:25.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803809 1-416611827715 2-73014444063 3-356482285578' 2026-02-25T22:32:25.090 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:25.090 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803809 2026-02-25T22:32:25.091 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:25.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:32:25.092 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803809 2026-02-25T22:32:25.092 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:25.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803809 2026-02-25T22:32:25.094 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2026-02-25T22:32:25.094 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803809 2026-02-25T22:32:25.094 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:32:25.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803807 -lt 25769803809 2026-02-25T22:32:25.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:32:26.311 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:32:26.312 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:32:26.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803807 -lt 25769803809 2026-02-25T22:32:26.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:32:27.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:32:27.523 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:32:27.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803809 -lt 25769803809 2026-02-25T22:32:27.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:27.732 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-416611827715 2026-02-25T22:32:27.732 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:27.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:32:27.734 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-416611827715 2026-02-25T22:32:27.734 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:27.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=416611827715 2026-02-25T22:32:27.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 416611827715' 2026-02-25T22:32:27.736 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 416611827715 2026-02-25T22:32:27.736 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:32:27.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 416611827715 -lt 416611827715 2026-02-25T22:32:27.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:27.949 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444063 2026-02-25T22:32:27.950 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:27.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:32:27.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444063 2026-02-25T22:32:27.952 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:27.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444063 2026-02-25T22:32:27.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444063' 2026-02-25T22:32:27.953 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 73014444063 2026-02-25T22:32:27.954 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:32:28.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444064 -lt 73014444063 2026-02-25T22:32:28.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:28.172 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-356482285578 2026-02-25T22:32:28.173 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:28.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:32:28.175 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-356482285578 2026-02-25T22:32:28.175 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:28.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=356482285578 2026-02-25T22:32:28.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 356482285578' 2026-02-25T22:32:28.176 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 356482285578 2026-02-25T22:32:28.177 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:32:28.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 356482285578 -lt 356482285578 2026-02-25T22:32:28.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:32:28.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:28.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:28.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:32:28.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:28.669 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:28.669 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:28.669 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:28.669 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:28.669 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:28.669 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:28.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:32:28.874 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:28.875 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:28.875 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:29.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:32:29.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:32:29.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:32:29.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:32:29.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:32:29.474 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:32:29.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2026-02-25T22:32:29.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-0-256 fail 2026-02-25T22:32:29.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:32:29.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:32:29.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-0-256 2026-02-25T22:32:29.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-25T22:32:29.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-25T22:32:29.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-73931-0-256 td/test-erasure-eio/COPY 2026-02-25T22:32:29.511 INFO:tasks.workunit.client.0.trial125.stderr:error getting pool-jerasure/obj-size-73931-0-256: (5) Input/output error 2026-02-25T22:32:29.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-25T22:32:29.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T22:32:29.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:332: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:32:29.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:32:29.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:32:29.729 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:32:29.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:32:29.999 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:32:30.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:32:30.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:32:30.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:32:30.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:32:30.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:32:30.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:32:30.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:32:30.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:32:30.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:32:30.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:32:30.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:32:30.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:32:30.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:32:30.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:32:30.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:32:30.123 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:32:30.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:32:30.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:32:30.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:32:30.125 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:32:30.126 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:32:30.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:32:30.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:32:30.151 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:32:30.151 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:30.151 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:30.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:32:30.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:32:30.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:32:30.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:32:30.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:32:30.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:32:30.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:32:30.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:32:30.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:32:30.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:32:30.154 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:32:30.154 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:32:30.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:32:30.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:32:30.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:32:30.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:32:30.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:32:30.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:32:30.158 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:32:30.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:32:30.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:32:30.159 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:32:30.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:32:30.160 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:32:30.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:32:30.161 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:32:30.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:32:30.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:32:30.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:32:30.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:32:30.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:30.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:30.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:32:30.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:32:30.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:32:30.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:32:30.168 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:32:30.168 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:30.168 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:30.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:32:30.170 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:32:30.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:32:30.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:32:30.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:32:30.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:32:30.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:32:30.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:32:30.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:32:30.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:32:30.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:32:30.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:32:30.192 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:32:30.192 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:32:30.192 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:32:30.193 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:32:30.193 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:30.193 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:30.193 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:30.193 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:32:30.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:32:30.224 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:32:30.224 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:32:30.224 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:32:30.224 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:32:30.225 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:32:30.226 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:32:30.226 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:32:30.226 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:32:30.226 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:32:30.226 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:30.226 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:30.226 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:32:30.227 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:32:30.227 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:32:30.282 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:32:30.282 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:32:30.282 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:32:30.282 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:32:30.283 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:32:30.283 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:32:30.283 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:32:30.284 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:32:30.284 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:32:30.284 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:30.284 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:30.284 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:32:30.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:32:30.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:32:30.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:32:30.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:32:30.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:32:30.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:32:30.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:32:30.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:32:30.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:32:30.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:32:30.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:32:30.449 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:32:30.450 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:32:30.450 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:30.450 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:30.450 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:30.451 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:32:30.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:32:30.471 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:32:30.472 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:32:30.578 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:32:30.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:32:31.589 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:32:31.589 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:32:31.590 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:32:31.590 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:32:31.590 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:31.590 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:31.590 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:32:31.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:32:31.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:32:31.639 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:32:31.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:32:31.646 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:32:30.207+0000 7ff0c988aec0 0 load: jerasure load: lrc 2026-02-25T22:32:31.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2026-02-25T22:32:31.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:336: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2026-02-25T22:32:31.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:337: TEST_rados_get_bad_size_shard_1: setup_osds 4 2026-02-25T22:32:31.648 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T22:32:31.648 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:32:31.648 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T22:32:31.648 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T22:32:31.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:32:31.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:32:31.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:32:31.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:32:31.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:32:31.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:32:31.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:32:31.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:32:31.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:32:31.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:32:31.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:32:31.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:32:31.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:32:31.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:32:31.655 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:32:31.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=77af5e3c-bf8f-4eee-a43d-49e42356acdd 2026-02-25T22:32:31.656 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 77af5e3c-bf8f-4eee-a43d-49e42356acdd 2026-02-25T22:32:31.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 77af5e3c-bf8f-4eee-a43d-49e42356acdd' 2026-02-25T22:32:31.657 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:32:31.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB/eJ9pDTCNJxAA9r+qRJQJNZYdY9Xn1aSSqA== 2026-02-25T22:32:31.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB/eJ9pDTCNJxAA9r+qRJQJNZYdY9Xn1aSSqA=="}' 2026-02-25T22:32:31.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 77af5e3c-bf8f-4eee-a43d-49e42356acdd -i td/test-erasure-eio/0/new.json 2026-02-25T22:32:31.775 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:32:31.783 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:32:31.784 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/eJ9pDTCNJxAA9r+qRJQJNZYdY9Xn1aSSqA== --osd-uuid 77af5e3c-bf8f-4eee-a43d-49e42356acdd 2026-02-25T22:32:31.800 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:31.795+0000 7f62debf0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:31.802 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:31.795+0000 7f62debf0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:31.803 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:31.795+0000 7f62debf0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:31.803 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:31.799+0000 7f62debf0a40 -1 bdev(0x55f25950a400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:32:31.804 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:31.799+0000 7f62debf0a40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:32:32.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:32:32.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:32:32.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:32:32.494 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:32:32.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:32:32.641 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:32:32.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:32:32.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:32:32.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:32:32.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:32:32.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:32:32.657 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:32.651+0000 7f8142466a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:32.659 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:32.655+0000 7f8142466a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:32.661 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:32.655+0000 7f8142466a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:32.752 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:32:32.752 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:32:32.752 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:32:32.752 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:32:32.752 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:32:32.752 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:32.753 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:32:32.753 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:32:32.753 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:32.753 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:32:32.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:32:33.117 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:33.111+0000 7f8142466a40 -1 Falling back to public interface 2026-02-25T22:32:33.706 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:33.699+0000 7f8142466a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:32:33.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:32:33.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:33.967 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:32:33.968 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:32:33.968 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:33.968 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:32:34.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:32:35.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:32:35.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:35.180 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:32:35.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:32:35.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:35.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:32:35.385 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2504229962,v1:127.0.0.1:6803/2504229962] [v2:127.0.0.1:6804/2504229962,v1:127.0.0.1:6805/2504229962] exists,up 77af5e3c-bf8f-4eee-a43d-49e42356acdd 2026-02-25T22:32:35.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:32:35.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:32:35.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:32:35.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:32:35.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:32:35.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:32:35.390 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:32:35.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:32:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:32:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:32:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:32:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:32:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:32:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:32:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:32:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:32:35.392 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:32:35.393 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:32:35.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5cca7305-4cb4-4010-acbe-483de2cd84ce 2026-02-25T22:32:35.393 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 5cca7305-4cb4-4010-acbe-483de2cd84ce 2026-02-25T22:32:35.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5cca7305-4cb4-4010-acbe-483de2cd84ce' 2026-02-25T22:32:35.394 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:32:35.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCDeJ9pzQrvFxAAe7kZwnA8EqHaEh3QqrmRGQ== 2026-02-25T22:32:35.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCDeJ9pzQrvFxAAe7kZwnA8EqHaEh3QqrmRGQ=="}' 2026-02-25T22:32:35.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5cca7305-4cb4-4010-acbe-483de2cd84ce -i td/test-erasure-eio/1/new.json 2026-02-25T22:32:35.628 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:32:35.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:32:35.640 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCDeJ9pzQrvFxAAe7kZwnA8EqHaEh3QqrmRGQ== --osd-uuid 5cca7305-4cb4-4010-acbe-483de2cd84ce 2026-02-25T22:32:35.662 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:35.655+0000 7f92e08d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:35.664 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:35.659+0000 7f92e08d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:35.665 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:35.659+0000 7f92e08d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:35.666 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:35.659+0000 7f92e08d3a40 -1 bdev(0x55bc10b5e400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:32:35.666 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:35.659+0000 7f92e08d3a40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:32:36.291 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:32:36.291 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:32:36.292 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:32:36.293 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:32:36.293 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:32:36.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:32:36.581 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:32:36.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:32:36.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:32:36.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:32:36.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:32:36.605 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:36.599+0000 7f252189aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:36.607 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:36.599+0000 7f252189aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:36.608 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:36.603+0000 7f252189aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:36.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:32:36.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:32:36.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:32:36.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:32:36.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:32:36.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:36.797 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:32:36.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:32:36.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:36.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:32:36.985 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:36.979+0000 7f252189aa40 -1 Falling back to public interface 2026-02-25T22:32:37.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:32:37.544 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:37.539+0000 7f252189aa40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:32:38.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:32:38.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:38.009 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:32:38.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:32:38.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:38.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:32:38.220 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2610137607,v1:127.0.0.1:6811/2610137607] [v2:127.0.0.1:6812/2610137607,v1:127.0.0.1:6813/2610137607] exists,up 5cca7305-4cb4-4010-acbe-483de2cd84ce 2026-02-25T22:32:38.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:32:38.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:32:38.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:32:38.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:32:38.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:32:38.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:32:38.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:32:38.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:32:38.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:32:38.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:32:38.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:32:38.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:32:38.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:32:38.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:32:38.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:38.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:32:38.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:32:38.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:32:38.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:32:38.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:32:38.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:32:38.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:32:38.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:32:38.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:32:38.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:32:38.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:32:38.226 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:32:38.227 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:32:38.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=95a7cf33-08b0-44c2-a70d-103d90060fdc 2026-02-25T22:32:38.227 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 95a7cf33-08b0-44c2-a70d-103d90060fdc 2026-02-25T22:32:38.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 95a7cf33-08b0-44c2-a70d-103d90060fdc' 2026-02-25T22:32:38.228 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:32:38.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCGeJ9pwpwjDhAAtnZqdjLaleTxH71nnDqyag== 2026-02-25T22:32:38.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCGeJ9pwpwjDhAAtnZqdjLaleTxH71nnDqyag=="}' 2026-02-25T22:32:38.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 95a7cf33-08b0-44c2-a70d-103d90060fdc -i td/test-erasure-eio/2/new.json 2026-02-25T22:32:38.453 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:32:38.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:32:38.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGeJ9pwpwjDhAAtnZqdjLaleTxH71nnDqyag== --osd-uuid 95a7cf33-08b0-44c2-a70d-103d90060fdc 2026-02-25T22:32:38.480 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:38.475+0000 7f495c162a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:38.482 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:38.475+0000 7f495c162a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:38.483 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:38.475+0000 7f495c162a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:38.483 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:38.479+0000 7f495c162a40 -1 bdev(0x55de887d4400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:32:38.484 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:38.479+0000 7f495c162a40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:32:39.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:32:39.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:32:39.139 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:32:39.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:32:39.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:32:39.421 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:32:39.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:32:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:32:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:32:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:32:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:32:39.443 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:39.435+0000 7f48d3c85a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:39.445 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:39.439+0000 7f48d3c85a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:39.446 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:39.439+0000 7f48d3c85a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:32:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:32:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:32:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:32:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:32:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:39.631 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:32:39.631 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:32:39.631 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:39.631 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:32:39.805 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:39.799+0000 7f48d3c85a40 -1 Falling back to public interface 2026-02-25T22:32:39.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:32:40.519 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:40.511+0000 7f48d3c85a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:32:40.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:32:40.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:40.841 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:32:40.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:32:40.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:40.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:32:41.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:32:42.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:32:42.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:42.055 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:32:42.055 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:32:42.055 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:42.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:32:42.261 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2604110298,v1:127.0.0.1:6819/2604110298] [v2:127.0.0.1:6820/2604110298,v1:127.0.0.1:6821/2604110298] exists,up 95a7cf33-08b0-44c2-a70d-103d90060fdc 2026-02-25T22:32:42.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:32:42.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:32:42.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:32:42.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:32:42.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:42.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:32:42.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:32:42.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:32:42.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:32:42.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:32:42.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:32:42.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:32:42.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:32:42.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:32:42.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:32:42.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:32:42.267 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:32:42.268 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:32:42.268 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cac91e54-286f-49ab-8221-8d565d66a9f0 2026-02-25T22:32:42.268 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 cac91e54-286f-49ab-8221-8d565d66a9f0 2026-02-25T22:32:42.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 cac91e54-286f-49ab-8221-8d565d66a9f0' 2026-02-25T22:32:42.269 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:32:42.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCKeJ9pgIaFEBAAYcZJSDrnv2ZNoy7nTLg/rw== 2026-02-25T22:32:42.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCKeJ9pgIaFEBAAYcZJSDrnv2ZNoy7nTLg/rw=="}' 2026-02-25T22:32:42.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cac91e54-286f-49ab-8221-8d565d66a9f0 -i td/test-erasure-eio/3/new.json 2026-02-25T22:32:42.487 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:32:42.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:32:42.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKeJ9pgIaFEBAAYcZJSDrnv2ZNoy7nTLg/rw== --osd-uuid cac91e54-286f-49ab-8221-8d565d66a9f0 2026-02-25T22:32:42.513 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:42.507+0000 7fac8b64ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:42.514 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:42.507+0000 7fac8b64ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:42.515 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:42.511+0000 7fac8b64ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:42.516 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:42.511+0000 7fac8b64ca40 -1 bdev(0x563f3c218400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:32:42.516 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:42.511+0000 7fac8b64ca40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:32:43.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:32:43.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:32:43.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:32:43.211 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:32:43.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:32:43.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:32:43.494 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:32:43.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:32:43.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:32:43.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:32:43.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:32:43.518 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:43.511+0000 7f372425fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:43.520 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:43.515+0000 7f372425fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:43.521 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:43.515+0000 7f372425fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:43.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:32:43.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:32:43.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:32:43.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:32:43.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:32:43.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:43.706 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:32:43.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:32:43.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:43.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:32:43.901 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:43.895+0000 7f372425fa40 -1 Falling back to public interface 2026-02-25T22:32:43.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:32:44.483 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:44.475+0000 7f372425fa40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:32:44.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:32:44.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:44.921 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:32:44.922 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:32:44.922 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:32:44.922 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:45.130 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1910098659,v1:127.0.0.1:6827/1910098659] [v2:127.0.0.1:6828/1910098659,v1:127.0.0.1:6829/1910098659] exists,up cac91e54-286f-49ab-8221-8d565d66a9f0 2026-02-25T22:32:45.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:32:45.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:32:45.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:32:45.131 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:32:45.132 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:32:45.132 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:32:45.132 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:32:45.133 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:45.133 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:45.133 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:32:45.133 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:32:45.133 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:32:45.187 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:32:45.193 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:32:45.194 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:32:33.115+0000 7f8142466a40 0 load: jerasure load: lrc 2026-02-25T22:32:45.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:339: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2026-02-25T22:32:45.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:340: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T22:32:45.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:32:45.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:32:45.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T22:32:45.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:32:45.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T22:32:45.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:32:45.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T22:32:45.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:32:45.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:32:45.849 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:32:45.859 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:32:46.860 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:32:46.860 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:32:46.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:32:46.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:32:46.861 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:32:46.862 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:32:46.862 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:32:46.862 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:32:46.863 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:32:46.863 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:32:46.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:32:46.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:32:46.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:32:46.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:32:46.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:32:46.977 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:32:47.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:32:47.186 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:47.186 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:32:47.186 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:32:47.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:32:47.186 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:47.186 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:32:47.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803781 2026-02-25T22:32:47.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803781 2026-02-25T22:32:47.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781' 2026-02-25T22:32:47.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:47.265 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:32:47.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-25T22:32:47.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-25T22:32:47.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-55834574852' 2026-02-25T22:32:47.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:47.344 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:32:47.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411332 2026-02-25T22:32:47.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411332 2026-02-25T22:32:47.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-55834574852 2-77309411332' 2026-02-25T22:32:47.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:47.422 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:32:47.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116995 2026-02-25T22:32:47.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116995 2026-02-25T22:32:47.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-55834574852 2-77309411332 3-115964116995' 2026-02-25T22:32:47.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:47.501 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803781 2026-02-25T22:32:47.501 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:47.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:32:47.502 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803781 2026-02-25T22:32:47.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:47.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803781 2026-02-25T22:32:47.504 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803781 2026-02-25T22:32:47.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803781' 2026-02-25T22:32:47.505 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:32:47.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803779 -lt 25769803781 2026-02-25T22:32:47.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:32:48.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:32:48.711 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:32:48.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803781 2026-02-25T22:32:48.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:48.925 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574852 2026-02-25T22:32:48.926 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:48.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:32:48.928 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574852 2026-02-25T22:32:48.928 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:48.929 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574852 2026-02-25T22:32:48.929 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574852 2026-02-25T22:32:48.930 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574852' 2026-02-25T22:32:48.930 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:32:49.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-25T22:32:49.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:49.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411332 2026-02-25T22:32:49.141 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:49.142 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:32:49.143 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411332 2026-02-25T22:32:49.143 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:49.144 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411332 2026-02-25T22:32:49.145 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 77309411332 2026-02-25T22:32:49.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411332' 2026-02-25T22:32:49.145 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:32:49.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411332 -lt 77309411332 2026-02-25T22:32:49.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:49.364 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116995 2026-02-25T22:32:49.364 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:49.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:32:49.366 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116995 2026-02-25T22:32:49.366 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:49.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116995 2026-02-25T22:32:49.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116995' 2026-02-25T22:32:49.368 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 115964116995 2026-02-25T22:32:49.368 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:32:49.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116995 -lt 115964116995 2026-02-25T22:32:49.575 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:32:49.575 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:49.576 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:49.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:32:49.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:49.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:49.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:49.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:49.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:49.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:49.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:50.054 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:32:50.055 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:50.055 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:50.055 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:50.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:32:50.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:32:50.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:32:50.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:342: TEST_rados_get_bad_size_shard_1: local shard_id=1 2026-02-25T22:32:50.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:343: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2026-02-25T22:32:50.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T22:32:50.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T22:32:50.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2026-02-25T22:32:50.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T22:32:50.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=10 2026-02-25T22:32:50.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T22:32:50.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2026-02-25T22:32:50.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T22:32:50.338 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-73931-1-10 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-73931-1-10 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-73931-1-10 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:32:50.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-73931-1-10 td/test-erasure-eio/ORIGINAL 2026-02-25T22:32:50.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-73931-1-10 td/test-erasure-eio 1 10 set 2026-02-25T22:32:50.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-1-10 2026-02-25T22:32:50.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:32:50.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:32:50.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:32:50.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T22:32:50.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:32:50.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2026-02-25T22:32:50.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:32:50.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T22:32:50.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:32:50.358 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-1-10 2026-02-25T22:32:50.358 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:32:50.358 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-1-10 2026-02-25T22:32:50.358 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-1-10 2026-02-25T22:32:50.359 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:32:50.563 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:32:50.563 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:50.564 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:32:50.564 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:32:50.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:32:50.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:32:50.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T22:32:50.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:32:50.894 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:32:50.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T22:32:50.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2026-02-25T22:32:50.919 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-25T22:32:50.919 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records in 2026-02-25T22:32:50.919 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records out 2026-02-25T22:32:50.919 INFO:tasks.workunit.client.0.trial125.stderr:10 bytes copied, 5.6104e-05 s, 178 kB/s 2026-02-25T22:32:50.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-73931-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:50.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:32:50.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:32:50.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T22:32:50.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:32:50.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-73931-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:50.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:32:50.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:32:50.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T22:32:50.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:32:50.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T22:32:50.921 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:32:50.921 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:32:50.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:32:50.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:32:50.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:32:51.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:32:51.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-73931-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:51.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:32:51.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:32:51.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T22:32:51.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:32:51.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T22:32:51.217 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-73931-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:51.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T22:32:51.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:32:51.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:32:51.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:32:51.526 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:32:51.527 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:32:51.527 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:32:51.527 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:32:51.527 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:51.527 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:51.527 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:51.529 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:51.529 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:32:51.529 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:32:51.529 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:32:51.529 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:32:51.529 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr:start osd.1 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:32:51.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:32:51.531 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:32:51.542 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:51.535+0000 7f6293ae5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:51.543 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:51.539+0000 7f6293ae5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:51.545 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:51.539+0000 7f6293ae5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:51.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T22:32:51.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:32:51.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:32:51.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:32:51.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:32:51.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:51.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:32:51.738 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:32:51.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:51.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:32:51.921 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:51.915+0000 7f6293ae5a40 -1 Falling back to public interface 2026-02-25T22:32:51.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:32:52.589 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:52.583+0000 7f6293ae5a40 -1 osd.1 36 log_to_monitors true 2026-02-25T22:32:52.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:32:52.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:52.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:32:52.948 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:52.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:52.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:32:53.153 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [13,36) [v2:127.0.0.1:6810/2532187700,v1:127.0.0.1:6811/2532187700] [v2:127.0.0.1:6812/2532187700,v1:127.0.0.1:6813/2532187700] exists,up 5cca7305-4cb4-4010-acbe-483de2cd84ce 2026-02-25T22:32:53.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:32:53.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:32:53.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:32:53.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:32:53.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:32:53.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:32:53.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:32:53.154 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:32:53.155 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:32:53.155 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:32:53.155 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:32:53.155 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:32:53.156 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:32:53.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:32:53.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:32:53.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:32:53.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:32:53.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:32:53.260 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:32:53.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:32:53.478 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:53.478 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:32:53.478 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:32:53.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:32:53.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:53.478 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:32:53.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803784 2026-02-25T22:32:53.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803784 2026-02-25T22:32:53.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784' 2026-02-25T22:32:53.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:53.556 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:32:53.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691843 2026-02-25T22:32:53.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691843 2026-02-25T22:32:53.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-171798691843' 2026-02-25T22:32:53.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:53.637 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:32:53.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411335 2026-02-25T22:32:53.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411335 2026-02-25T22:32:53.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-171798691843 2-77309411335' 2026-02-25T22:32:53.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:32:53.714 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:32:53.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116998 2026-02-25T22:32:53.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116998 2026-02-25T22:32:53.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-171798691843 2-77309411335 3-115964116998' 2026-02-25T22:32:53.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:53.794 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803784 2026-02-25T22:32:53.794 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:53.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:32:53.795 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803784 2026-02-25T22:32:53.795 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:53.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803784 2026-02-25T22:32:53.797 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-25T22:32:53.797 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803784 2026-02-25T22:32:53.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:32:54.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803784 2026-02-25T22:32:54.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:32:55.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:32:55.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:32:55.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803784 -lt 25769803784 2026-02-25T22:32:55.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:55.236 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-171798691843 2026-02-25T22:32:55.236 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:55.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:32:55.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-171798691843 2026-02-25T22:32:55.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:55.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691843 2026-02-25T22:32:55.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 171798691843' 2026-02-25T22:32:55.240 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 171798691843 2026-02-25T22:32:55.240 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:32:55.454 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691843 -lt 171798691843 2026-02-25T22:32:55.454 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:55.454 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411335 2026-02-25T22:32:55.455 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:55.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:32:55.457 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411335 2026-02-25T22:32:55.457 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:55.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411335 2026-02-25T22:32:55.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411335' 2026-02-25T22:32:55.459 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 77309411335 2026-02-25T22:32:55.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:32:55.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411335 -lt 77309411335 2026-02-25T22:32:55.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:32:55.667 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116998 2026-02-25T22:32:55.667 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:32:55.668 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:32:55.669 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116998 2026-02-25T22:32:55.669 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:32:55.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116998 2026-02-25T22:32:55.670 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116998' 2026-02-25T22:32:55.671 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 115964116998 2026-02-25T22:32:55.671 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:32:55.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116998 -lt 115964116998 2026-02-25T22:32:55.887 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:32:55.888 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:55.888 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:56.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:32:56.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:32:56.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:32:56.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:32:56.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:32:56.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:32:56.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:32:56.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:32:56.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:32:56.386 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:32:56.386 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:32:56.386 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:32:56.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:32:56.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:32:56.662 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:32:56.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:32:56.663 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:32:56.933 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:32:56.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-1-10 2026-02-25T22:32:56.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:32:56.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:32:56.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-1-10 2026-02-25T22:32:56.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:32:56.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:32:56.945 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-73931-1-10 td/test-erasure-eio/COPY 2026-02-25T22:32:56.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:32:56.966 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:32:56.967 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2026-02-25T22:32:56.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2026-02-25T22:32:56.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-73931-1-10 td/test-erasure-eio 2 10 set 2026-02-25T22:32:56.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-1-10 2026-02-25T22:32:56.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:32:56.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:32:56.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:32:56.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2026-02-25T22:32:56.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:32:56.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2026-02-25T22:32:56.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:32:56.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T22:32:56.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:32:56.971 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-1-10 2026-02-25T22:32:56.971 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:32:56.971 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-1-10 2026-02-25T22:32:56.971 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-1-10 2026-02-25T22:32:56.972 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:32:57.178 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:32:57.178 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:57.178 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:32:57.178 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:32:57.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:32:57.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:32:57.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2026-02-25T22:32:57.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:32:57.441 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:32:57.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T22:32:57.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2026-02-25T22:32:57.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-25T22:32:57.454 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records in 2026-02-25T22:32:57.454 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records out 2026-02-25T22:32:57.454 INFO:tasks.workunit.client.0.trial125.stderr:10 bytes copied, 5.8929e-05 s, 170 kB/s 2026-02-25T22:32:57.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-73931-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:57.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:32:57.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:32:57.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-25T22:32:57.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:32:57.456 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-73931-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:57.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:32:57.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:32:57.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-25T22:32:57.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:32:57.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T22:32:57.457 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:32:57.457 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:32:57.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:32:57.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:32:57.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:32:57.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:32:57.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-73931-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:57.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:32:57.765 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:32:57.765 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-25T22:32:57.765 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:32:57.765 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-25T22:32:57.765 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-73931-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:32:58.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-25T22:32:58.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:32:58.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:32:58.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T22:32:58.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:32:58.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:32:58.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:32:58.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:32:58.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:32:58.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:32:58.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:32:58.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:32:58.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:32:58.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:32:58.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:32:58.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:32:58.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:32:58.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:32:58.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:32:58.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:32:58.063 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:32:58.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:58.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:32:58.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:32:58.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:32:58.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:32:58.065 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:32:58.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:32:58.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:32:58.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:32:58.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:32:58.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:32:58.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:32:58.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T22:32:58.066 INFO:tasks.workunit.client.0.trial125.stderr:start osd.2 2026-02-25T22:32:58.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:32:58.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T22:32:58.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T22:32:58.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:32:58.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:32:58.067 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:32:58.082 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:58.079+0000 7ff66826da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:58.084 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:58.083+0000 7ff66826da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:58.085 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:58.083+0000 7ff66826da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:32:58.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T22:32:58.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:32:58.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:32:58.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:32:58.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:32:58.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:58.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:32:58.282 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:32:58.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:58.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:32:58.425 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:58.423+0000 7ff66826da40 -1 Falling back to public interface 2026-02-25T22:32:58.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:32:59.106 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:32:59.103+0000 7ff66826da40 -1 osd.2 45 log_to_monitors true 2026-02-25T22:32:59.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:32:59.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:32:59.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:32:59.500 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:32:59.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:32:59.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:32:59.718 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 up in weight 1 up_from 51 up_thru 37 down_at 48 last_clean_interval [18,45) [v2:127.0.0.1:6818/805958887,v1:127.0.0.1:6819/805958887] [v2:127.0.0.1:6820/805958887,v1:127.0.0.1:6821/805958887] exists,up 95a7cf33-08b0-44c2-a70d-103d90060fdc 2026-02-25T22:32:59.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:32:59.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:32:59.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:32:59.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:32:59.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:32:59.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:32:59.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:32:59.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:32:59.719 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:32:59.719 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:32:59.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:32:59.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:32:59.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:32:59.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:32:59.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:32:59.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:32:59.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:32:59.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:32:59.828 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:33:00.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:33:00.038 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:00.038 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:33:00.038 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:33:00.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:33:00.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:00.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:33:00.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803788 2026-02-25T22:33:00.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803788 2026-02-25T22:33:00.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788' 2026-02-25T22:33:00.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:00.118 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:33:00.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691846 2026-02-25T22:33:00.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691846 2026-02-25T22:33:00.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-171798691846' 2026-02-25T22:33:00.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:00.197 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:33:00.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=219043332099 2026-02-25T22:33:00.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 219043332099 2026-02-25T22:33:00.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-171798691846 2-219043332099' 2026-02-25T22:33:00.276 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:00.276 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:33:00.354 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117001 2026-02-25T22:33:00.354 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117001 2026-02-25T22:33:00.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-171798691846 2-219043332099 3-115964117001' 2026-02-25T22:33:00.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:00.355 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803788 2026-02-25T22:33:00.355 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:00.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:33:00.357 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803788 2026-02-25T22:33:00.357 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:00.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803788 2026-02-25T22:33:00.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803788' 2026-02-25T22:33:00.359 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803788 2026-02-25T22:33:00.359 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:00.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803788 2026-02-25T22:33:00.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:33:01.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:33:01.582 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:01.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803788 -lt 25769803788 2026-02-25T22:33:01.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:01.795 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-171798691846 2026-02-25T22:33:01.795 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:01.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:33:01.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-171798691846 2026-02-25T22:33:01.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:01.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691846 2026-02-25T22:33:01.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2026-02-25T22:33:01.798 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 171798691846 2026-02-25T22:33:01.799 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:33:02.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691846 -lt 171798691846 2026-02-25T22:33:02.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:02.009 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-219043332099 2026-02-25T22:33:02.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:02.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:33:02.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-219043332099 2026-02-25T22:33:02.012 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:02.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=219043332099 2026-02-25T22:33:02.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 219043332099' 2026-02-25T22:33:02.013 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 219043332099 2026-02-25T22:33:02.014 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:33:02.229 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 219043332099 -lt 219043332099 2026-02-25T22:33:02.229 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:02.230 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117001 2026-02-25T22:33:02.230 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:02.232 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:33:02.232 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117001 2026-02-25T22:33:02.232 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:02.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117001 2026-02-25T22:33:02.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117001' 2026-02-25T22:33:02.234 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 115964117001 2026-02-25T22:33:02.234 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:33:02.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117002 -lt 115964117001 2026-02-25T22:33:02.444 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:33:02.445 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:02.445 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:02.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:33:02.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:33:02.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:33:02.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:33:02.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:33:02.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:33:02.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:33:02.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:33:02.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:33:02.926 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:33:02.926 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:02.926 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:03.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:33:03.208 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:33:03.208 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:33:03.208 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:33:03.208 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:33:03.523 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:33:03.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2026-02-25T22:33:03.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-1-10 2026-02-25T22:33:03.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:33:03.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:33:03.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-1-10 2026-02-25T22:33:03.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:33:03.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:33:03.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-73931-1-10 td/test-erasure-eio/COPY 2026-02-25T22:33:03.554 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:33:03.554 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:33:03.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T22:33:03.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:344: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 0 2026-02-25T22:33:03.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T22:33:03.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T22:33:03.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2026-02-25T22:33:03.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T22:33:03.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=0 2026-02-25T22:33:03.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T22:33:03.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2026-02-25T22:33:03.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T22:33:03.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-73931-1-0 2026-02-25T22:33:03.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-73931-1-0 2026-02-25T22:33:03.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:33:03.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:33:03.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-73931-1-0 2026-02-25T22:33:03.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:33:03.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:33:03.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:33:03.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:33:03.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:33:03.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:33:03.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:33:03.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:33:03.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-73931-1-0 td/test-erasure-eio/ORIGINAL 2026-02-25T22:33:03.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-73931-1-0 td/test-erasure-eio 1 0 set 2026-02-25T22:33:03.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-1-0 2026-02-25T22:33:03.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:33:03.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:33:03.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:33:03.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T22:33:03.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:33:03.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2026-02-25T22:33:03.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:33:03.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T22:33:03.584 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:33:03.584 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-1-0 2026-02-25T22:33:03.585 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:33:03.585 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-1-0 2026-02-25T22:33:03.585 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-1-0 2026-02-25T22:33:03.585 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:33:03.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:33:03.797 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:03.797 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:33:03.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:33:03.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:33:03.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:33:03.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T22:33:03.798 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:33:04.088 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:33:04.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T22:33:04.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2026-02-25T22:33:04.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-25T22:33:04.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-73931-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:04.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:33:04.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:33:04.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T22:33:04.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:33:04.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-73931-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:04.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:33:04.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:33:04.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T22:33:04.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:33:04.103 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T22:33:04.104 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:33:04.104 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:33:04.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:33:04.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:33:04.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:33:04.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:33:04.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-73931-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:04.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:33:04.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:33:04.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T22:33:04.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:33:04.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T22:33:04.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-73931-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:04.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T22:33:04.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:33:04.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:33:04.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T22:33:04.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:33:04.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:33:04.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:33:04.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:33:04.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:33:04.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:33:04.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:33:04.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:33:04.722 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:33:04.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:33:04.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:33:04.723 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:33:04.723 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:33:04.723 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:33:04.723 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:33:04.723 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:04.723 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:04.723 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:04.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:04.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:33:04.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr:start osd.1 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:33:04.726 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T22:33:04.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T22:33:04.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:33:04.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:33:04.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:33:04.745 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:04.742+0000 7f0f53a0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:04.747 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:04.742+0000 7f0f53a0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:04.748 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:04.746+0000 7f0f53a0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:04.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T22:33:04.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:33:04.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:33:04.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:33:04.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:33:04.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:04.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:33:04.933 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:33:04.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:04.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:33:05.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:33:05.197 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:05.194+0000 7f0f53a0ea40 -1 Falling back to public interface 2026-02-25T22:33:05.797 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:05.794+0000 7f0f53a0ea40 -1 osd.1 56 log_to_monitors true 2026-02-25T22:33:06.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:33:06.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:06.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:33:06.149 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:06.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:06.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:33:06.364 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 up in weight 1 up_from 60 up_thru 60 down_at 57 last_clean_interval [40,56) [v2:127.0.0.1:6810/2826991624,v1:127.0.0.1:6811/2826991624] [v2:127.0.0.1:6812/2826991624,v1:127.0.0.1:6813/2826991624] exists,up 5cca7305-4cb4-4010-acbe-483de2cd84ce 2026-02-25T22:33:06.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:33:06.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:33:06.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:33:06.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:33:06.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:33:06.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:33:06.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:33:06.365 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:33:06.365 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:33:06.365 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:33:06.366 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:33:06.366 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:33:06.366 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:33:06.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:33:06.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:33:06.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:33:06.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:33:06.467 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:33:06.467 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:33:06.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:33:06.681 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:06.681 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:33:06.681 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:33:06.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:33:06.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:06.681 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:33:06.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803791 2026-02-25T22:33:06.760 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803791 2026-02-25T22:33:06.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791' 2026-02-25T22:33:06.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:06.761 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:33:06.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037763 2026-02-25T22:33:06.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037763 2026-02-25T22:33:06.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-257698037763' 2026-02-25T22:33:06.841 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:06.841 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:33:06.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=219043332102 2026-02-25T22:33:06.920 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 219043332102 2026-02-25T22:33:06.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-257698037763 2-219043332102' 2026-02-25T22:33:06.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:06.921 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:33:06.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117005 2026-02-25T22:33:06.997 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117005 2026-02-25T22:33:06.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-257698037763 2-219043332102 3-115964117005' 2026-02-25T22:33:06.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:06.998 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803791 2026-02-25T22:33:06.998 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:06.999 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:33:07.000 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803791 2026-02-25T22:33:07.000 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:07.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803791 2026-02-25T22:33:07.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803791' 2026-02-25T22:33:07.002 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803791 2026-02-25T22:33:07.002 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:07.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803789 -lt 25769803791 2026-02-25T22:33:07.215 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:33:08.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:33:08.217 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:08.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803789 -lt 25769803791 2026-02-25T22:33:08.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:33:09.429 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:33:09.430 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:09.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803791 -lt 25769803791 2026-02-25T22:33:09.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:09.640 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-257698037763 2026-02-25T22:33:09.640 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:09.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:33:09.642 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-257698037763 2026-02-25T22:33:09.642 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:09.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=257698037763 2026-02-25T22:33:09.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 257698037763' 2026-02-25T22:33:09.644 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 257698037763 2026-02-25T22:33:09.644 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:33:09.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 257698037763 -lt 257698037763 2026-02-25T22:33:09.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:09.858 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-219043332102 2026-02-25T22:33:09.858 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:09.860 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:33:09.860 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-219043332102 2026-02-25T22:33:09.860 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:09.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=219043332102 2026-02-25T22:33:09.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 219043332102' 2026-02-25T22:33:09.862 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 219043332102 2026-02-25T22:33:09.862 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:33:10.080 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 219043332102 -lt 219043332102 2026-02-25T22:33:10.080 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:10.081 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117005 2026-02-25T22:33:10.081 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:10.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:33:10.083 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117005 2026-02-25T22:33:10.083 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:10.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117005 2026-02-25T22:33:10.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117005' 2026-02-25T22:33:10.085 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 115964117005 2026-02-25T22:33:10.085 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:33:10.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117005 -lt 115964117005 2026-02-25T22:33:10.307 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:33:10.307 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:10.307 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:10.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:33:10.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:33:10.582 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:33:10.582 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:33:10.582 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:33:10.582 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:33:10.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:33:10.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:33:10.783 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:33:10.784 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:33:10.784 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:10.784 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:11.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:33:11.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:33:11.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:33:11.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:33:11.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:33:11.380 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:33:11.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-1-0 2026-02-25T22:33:11.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:33:11.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:33:11.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-1-0 2026-02-25T22:33:11.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:33:11.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:33:11.395 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-73931-1-0 td/test-erasure-eio/COPY 2026-02-25T22:33:11.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:33:11.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:33:11.425 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2026-02-25T22:33:11.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2026-02-25T22:33:11.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-73931-1-0 td/test-erasure-eio 2 0 set 2026-02-25T22:33:11.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-1-0 2026-02-25T22:33:11.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:33:11.427 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-1-0 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-1-0 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-1-0 2026-02-25T22:33:11.428 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:33:11.635 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:33:11.636 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:11.636 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:33:11.636 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:33:11.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:33:11.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:33:11.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2026-02-25T22:33:11.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:33:11.948 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:33:11.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T22:33:11.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2026-02-25T22:33:11.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-25T22:33:11.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-73931-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:11.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:33:11.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:33:11.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-25T22:33:11.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:33:11.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-73931-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:11.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:33:11.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:33:11.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-25T22:33:11.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:33:11.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T22:33:11.964 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:33:11.964 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:33:11.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:33:11.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:33:11.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:33:12.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:33:12.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-73931-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:12.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:33:12.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:33:12.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-25T22:33:12.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:33:12.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-25T22:33:12.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-73931-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:12.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-25T22:33:12.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:33:12.581 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:33:12.582 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:33:12.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:33:12.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:33:12.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:33:12.583 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:33:12.583 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:12.583 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:12.583 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:12.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:12.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:33:12.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:33:12.585 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr:start osd.2 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T22:33:12.586 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T22:33:12.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:33:12.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:33:12.587 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:33:12.601 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:12.598+0000 7f445d8a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:12.602 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:12.598+0000 7f445d8a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:12.603 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:12.602+0000 7f445d8a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:12.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T22:33:12.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:33:12.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:33:12.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:33:12.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:33:12.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:12.800 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:33:12.800 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:33:12.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:12.801 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:33:13.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:33:13.093 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:13.090+0000 7f445d8a2a40 -1 Falling back to public interface 2026-02-25T22:33:13.746 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:13.742+0000 7f445d8a2a40 -1 osd.2 64 log_to_monitors true 2026-02-25T22:33:14.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:33:14.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:14.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:33:14.008 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:14.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:14.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:33:14.211 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 up in weight 1 up_from 69 up_thru 57 down_at 66 last_clean_interval [51,64) [v2:127.0.0.1:6818/3309900901,v1:127.0.0.1:6819/3309900901] [v2:127.0.0.1:6820/3309900901,v1:127.0.0.1:6821/3309900901] exists,up 95a7cf33-08b0-44c2-a70d-103d90060fdc 2026-02-25T22:33:14.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:33:14.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:33:14.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:33:14.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:33:14.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:33:14.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:33:14.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:33:14.213 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:33:14.214 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:33:14.214 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:33:14.214 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:33:14.214 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:33:14.214 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:33:14.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:33:14.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:33:14.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:33:14.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:33:14.328 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:33:14.328 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:33:14.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:33:14.543 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:14.543 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:33:14.543 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:33:14.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:33:14.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:14.543 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:33:14.620 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803794 2026-02-25T22:33:14.620 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803794 2026-02-25T22:33:14.620 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794' 2026-02-25T22:33:14.620 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:14.621 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:33:14.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037766 2026-02-25T22:33:14.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037766 2026-02-25T22:33:14.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794 1-257698037766' 2026-02-25T22:33:14.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:14.701 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:33:14.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743427 2026-02-25T22:33:14.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743427 2026-02-25T22:33:14.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794 1-257698037766 2-296352743427' 2026-02-25T22:33:14.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:14.778 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:33:14.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117008 2026-02-25T22:33:14.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117008 2026-02-25T22:33:14.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794 1-257698037766 2-296352743427 3-115964117008' 2026-02-25T22:33:14.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:14.858 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803794 2026-02-25T22:33:14.858 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:14.859 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:33:14.859 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803794 2026-02-25T22:33:14.860 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:14.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803794 2026-02-25T22:33:14.861 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803794' 2026-02-25T22:33:14.862 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803794 2026-02-25T22:33:14.862 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:15.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803794 -lt 25769803794 2026-02-25T22:33:15.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:15.074 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-257698037766 2026-02-25T22:33:15.074 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:15.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:33:15.077 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-257698037766 2026-02-25T22:33:15.077 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:15.078 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=257698037766 2026-02-25T22:33:15.078 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 257698037766' 2026-02-25T22:33:15.078 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 257698037766 2026-02-25T22:33:15.079 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:33:15.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 257698037764 -lt 257698037766 2026-02-25T22:33:15.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:33:16.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:33:16.301 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:33:16.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 257698037764 -lt 257698037766 2026-02-25T22:33:16.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:33:17.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:33:17.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:33:17.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 257698037767 -lt 257698037766 2026-02-25T22:33:17.730 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:17.731 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-296352743427 2026-02-25T22:33:17.731 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:17.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:33:17.733 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743427 2026-02-25T22:33:17.733 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:17.734 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=296352743427 2026-02-25T22:33:17.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 296352743427' 2026-02-25T22:33:17.735 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 296352743427 2026-02-25T22:33:17.735 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:33:17.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 296352743427 -lt 296352743427 2026-02-25T22:33:17.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:17.961 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117008 2026-02-25T22:33:17.961 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:17.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:33:17.963 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117008 2026-02-25T22:33:17.963 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:17.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117008 2026-02-25T22:33:17.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117008' 2026-02-25T22:33:17.965 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 115964117008 2026-02-25T22:33:17.965 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:33:18.179 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117009 -lt 115964117008 2026-02-25T22:33:18.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:33:18.180 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:18.180 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:18.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:33:18.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:33:18.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:33:18.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:33:18.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:33:18.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:33:18.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:33:18.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:33:18.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:33:18.668 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:33:18.668 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:18.669 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:18.954 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:33:18.954 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:33:18.954 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:33:18.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:33:18.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:33:19.268 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:33:19.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2026-02-25T22:33:19.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-1-0 2026-02-25T22:33:19.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:33:19.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:33:19.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-1-0 2026-02-25T22:33:19.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:33:19.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:33:19.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-73931-1-0 td/test-erasure-eio/COPY 2026-02-25T22:33:19.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:33:19.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:33:19.308 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T22:33:19.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:345: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 256 add 2026-02-25T22:33:19.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T22:33:19.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T22:33:19.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2026-02-25T22:33:19.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T22:33:19.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=256 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=add 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-73931-1-256 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-73931-1-256 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-73931-1-256 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:33:19.313 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:33:19.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:33:19.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:33:19.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:33:19.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-73931-1-256 td/test-erasure-eio/ORIGINAL 2026-02-25T22:33:19.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-73931-1-256 td/test-erasure-eio 1 256 add 2026-02-25T22:33:19.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-1-256 2026-02-25T22:33:19.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:33:19.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:33:19.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:33:19.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T22:33:19.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:33:19.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2026-02-25T22:33:19.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:33:19.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2026-02-25T22:33:19.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:33:19.336 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-1-256 2026-02-25T22:33:19.336 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:33:19.336 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-1-256 2026-02-25T22:33:19.336 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-1-256 2026-02-25T22:33:19.336 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:33:19.546 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:33:19.546 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:19.547 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:33:19.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:33:19.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:33:19.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:33:19.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T22:33:19.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:33:19.831 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:33:19.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2026-02-25T22:33:19.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-73931-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:19.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:33:19.843 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-73931-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:33:19.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:33:19.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:33:20.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:33:20.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-73931-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:20.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:33:20.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:33:20.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T22:33:20.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:33:20.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T22:33:20.153 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-73931-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:20.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T22:33:20.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:33:20.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:33:20.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T22:33:20.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:33:20.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:33:20.325 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:33:20.326 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:33:20.327 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:20.327 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:20.327 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:20.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:20.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:33:20.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:33:20.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:33:20.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:33:20.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:33:20.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:33:20.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:33:20.329 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:33:20.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:33:20.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:33:20.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T22:33:20.330 INFO:tasks.workunit.client.0.trial125.stderr:start osd.1 2026-02-25T22:33:20.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:33:20.330 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T22:33:20.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T22:33:20.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:33:20.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:33:20.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:33:20.346 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:20.342+0000 7f8357d2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:20.347 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:20.342+0000 7f8357d2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:20.348 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:20.346+0000 7f8357d2ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:20.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T22:33:20.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:33:20.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:33:20.541 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:33:20.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:33:20.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:20.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:33:20.542 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:33:20.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:20.542 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:33:20.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:33:20.821 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:20.818+0000 7f8357d2ea40 -1 Falling back to public interface 2026-02-25T22:33:21.471 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:21.466+0000 7f8357d2ea40 -1 osd.1 74 log_to_monitors true 2026-02-25T22:33:21.755 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:33:21.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:21.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:33:21.756 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:21.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:21.756 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:33:21.960 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [60,74) [v2:127.0.0.1:6810/3878723063,v1:127.0.0.1:6811/3878723063] [v2:127.0.0.1:6812/3878723063,v1:127.0.0.1:6813/3878723063] exists,up 5cca7305-4cb4-4010-acbe-483de2cd84ce 2026-02-25T22:33:21.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:33:21.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:33:21.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:33:21.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:33:21.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:33:21.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:33:21.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:33:21.961 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:33:21.962 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:33:21.962 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:33:21.962 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:33:21.962 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:33:21.962 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:33:22.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:33:22.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:33:22.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:33:22.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:33:22.069 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:33:22.069 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:33:22.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:33:22.280 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:22.280 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:33:22.280 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:33:22.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:33:22.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:22.281 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:33:22.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803798 2026-02-25T22:33:22.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803798 2026-02-25T22:33:22.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803798' 2026-02-25T22:33:22.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:22.360 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:33:22.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=335007449091 2026-02-25T22:33:22.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 335007449091 2026-02-25T22:33:22.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803798 1-335007449091' 2026-02-25T22:33:22.438 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:22.439 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:33:22.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743430 2026-02-25T22:33:22.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743430 2026-02-25T22:33:22.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803798 1-335007449091 2-296352743430' 2026-02-25T22:33:22.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:22.517 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:33:22.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117012 2026-02-25T22:33:22.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117012 2026-02-25T22:33:22.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803798 1-335007449091 2-296352743430 3-115964117012' 2026-02-25T22:33:22.593 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:22.594 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803798 2026-02-25T22:33:22.594 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:22.595 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:33:22.596 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803798 2026-02-25T22:33:22.596 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:22.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803798 2026-02-25T22:33:22.597 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803798' 2026-02-25T22:33:22.598 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803798 2026-02-25T22:33:22.598 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803798 -lt 25769803798 2026-02-25T22:33:22.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:22.819 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-335007449091 2026-02-25T22:33:22.819 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:22.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:33:22.821 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-335007449091 2026-02-25T22:33:22.821 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:22.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=335007449091 2026-02-25T22:33:22.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 335007449091' 2026-02-25T22:33:22.823 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 335007449091 2026-02-25T22:33:22.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:33:23.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 335007449091 -lt 335007449091 2026-02-25T22:33:23.038 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:23.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-296352743430 2026-02-25T22:33:23.039 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:23.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:33:23.041 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743430 2026-02-25T22:33:23.041 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:23.042 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=296352743430 2026-02-25T22:33:23.043 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 296352743430' 2026-02-25T22:33:23.043 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 296352743430 2026-02-25T22:33:23.043 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:33:23.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 296352743430 -lt 296352743430 2026-02-25T22:33:23.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:23.265 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117012 2026-02-25T22:33:23.265 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:23.266 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:33:23.267 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117012 2026-02-25T22:33:23.267 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:23.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117012 2026-02-25T22:33:23.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117012' 2026-02-25T22:33:23.269 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 115964117012 2026-02-25T22:33:23.269 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:33:23.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117012 -lt 115964117012 2026-02-25T22:33:23.488 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:33:23.489 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:23.489 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:23.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:33:23.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:33:23.763 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:33:23.763 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:33:23.763 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:33:23.763 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:33:23.763 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:33:23.763 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:33:23.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:33:23.965 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:33:23.965 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:23.965 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:24.236 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:33:24.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:33:24.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:33:24.237 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-25T22:33:24.237 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records in 2026-02-25T22:33:24.237 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records out 2026-02-25T22:33:24.237 INFO:tasks.workunit.client.0.trial125.stderr:256 bytes copied, 2.5187e-05 s, 10.2 MB/s 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-73931-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-73931-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:33:24.238 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:33:24.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:33:24.239 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:33:24.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:33:24.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-73931-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:24.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:33:24.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:33:24.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T22:33:24.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:33:24.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T22:33:24.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-73931-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:33:24.816 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:33:24.817 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:33:24.817 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:24.817 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:24.817 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:24.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:24.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:33:24.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:33:24.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:33:24.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:33:24.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:33:24.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:33:24.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:33:24.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:33:24.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:33:24.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:33:24.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T22:33:24.819 INFO:tasks.workunit.client.0.trial125.stderr:start osd.1 2026-02-25T22:33:24.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:33:24.820 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T22:33:24.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T22:33:24.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:33:24.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:33:24.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:33:24.835 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:24.834+0000 7f4c6a3c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:24.837 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:24.834+0000 7f4c6a3c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:24.838 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:24.834+0000 7f4c6a3c1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:25.034 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T22:33:25.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:33:25.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:33:25.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:33:25.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:33:25.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:25.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:33:25.035 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:33:25.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:25.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:33:25.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:33:25.305 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:25.302+0000 7f4c6a3c1a40 -1 Falling back to public interface 2026-02-25T22:33:25.984 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:25.982+0000 7f4c6a3c1a40 -1 osd.1 79 log_to_monitors true 2026-02-25T22:33:26.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:33:26.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:26.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:33:26.248 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:26.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:26.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:33:26.458 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [78,79) [v2:127.0.0.1:6810/3401246112,v1:127.0.0.1:6811/3401246112] [v2:127.0.0.1:6812/3401246112,v1:127.0.0.1:6813/3401246112] exists,up 5cca7305-4cb4-4010-acbe-483de2cd84ce 2026-02-25T22:33:26.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:33:26.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:33:26.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:33:26.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:33:26.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:33:26.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:33:26.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:33:26.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:33:26.459 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:33:26.459 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:33:26.460 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:33:26.460 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:33:26.460 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:33:26.569 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:33:26.569 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:33:26.569 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:33:26.569 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:33:26.569 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:33:26.570 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:33:26.786 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:33:26.786 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:26.786 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:33:26.786 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:33:26.786 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:33:26.786 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:26.787 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:33:26.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803801 2026-02-25T22:33:26.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803801 2026-02-25T22:33:26.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801' 2026-02-25T22:33:26.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:26.864 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:33:26.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285571 2026-02-25T22:33:26.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285571 2026-02-25T22:33:26.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801 1-356482285571' 2026-02-25T22:33:26.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:26.942 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:33:27.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743433 2026-02-25T22:33:27.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743433 2026-02-25T22:33:27.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801 1-356482285571 2-296352743433' 2026-02-25T22:33:27.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:27.020 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:33:27.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117015 2026-02-25T22:33:27.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117015 2026-02-25T22:33:27.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801 1-356482285571 2-296352743433 3-115964117015' 2026-02-25T22:33:27.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:27.100 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803801 2026-02-25T22:33:27.100 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:27.101 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:33:27.102 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803801 2026-02-25T22:33:27.102 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:27.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803801 2026-02-25T22:33:27.104 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2026-02-25T22:33:27.104 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803801 2026-02-25T22:33:27.104 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:27.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803799 -lt 25769803801 2026-02-25T22:33:27.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:33:28.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:33:28.317 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:28.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803799 -lt 25769803801 2026-02-25T22:33:28.530 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:33:29.531 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:33:29.532 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:29.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803801 -lt 25769803801 2026-02-25T22:33:29.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:29.741 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-356482285571 2026-02-25T22:33:29.741 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:29.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:33:29.744 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-356482285571 2026-02-25T22:33:29.744 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:29.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=356482285571 2026-02-25T22:33:29.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 356482285571' 2026-02-25T22:33:29.745 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 356482285571 2026-02-25T22:33:29.746 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:33:29.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 356482285571 -lt 356482285571 2026-02-25T22:33:29.954 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:29.954 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-296352743433 2026-02-25T22:33:29.955 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:29.956 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:33:29.956 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743433 2026-02-25T22:33:29.957 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:29.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=296352743433 2026-02-25T22:33:29.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 296352743433' 2026-02-25T22:33:29.958 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 296352743433 2026-02-25T22:33:29.958 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:33:30.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 296352743433 -lt 296352743433 2026-02-25T22:33:30.176 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:30.176 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117015 2026-02-25T22:33:30.177 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:30.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:33:30.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117015 2026-02-25T22:33:30.179 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:30.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117015 2026-02-25T22:33:30.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117015' 2026-02-25T22:33:30.180 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 115964117015 2026-02-25T22:33:30.181 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:33:30.399 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117015 -lt 115964117015 2026-02-25T22:33:30.399 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:33:30.400 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:30.400 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:30.675 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:33:30.676 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:33:30.676 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:33:30.677 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:33:30.677 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:33:30.677 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:33:30.677 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:33:30.677 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:33:30.896 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:33:30.896 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:33:30.897 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:30.897 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:31.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:33:31.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:33:31.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:33:31.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:33:31.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:33:31.487 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:33:31.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-1-256 2026-02-25T22:33:31.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:33:31.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:33:31.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-1-256 2026-02-25T22:33:31.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:33:31.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:33:31.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-73931-1-256 td/test-erasure-eio/COPY 2026-02-25T22:33:31.519 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:33:31.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:33:31.522 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2026-02-25T22:33:31.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2026-02-25T22:33:31.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-73931-1-256 td/test-erasure-eio 2 256 add 2026-02-25T22:33:31.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-73931-1-256 2026-02-25T22:33:31.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T22:33:31.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T22:33:31.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T22:33:31.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2026-02-25T22:33:31.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T22:33:31.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2026-02-25T22:33:31.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T22:33:31.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2026-02-25T22:33:31.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T22:33:31.525 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-73931-1-256 2026-02-25T22:33:31.525 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:33:31.525 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-73931-1-256 2026-02-25T22:33:31.525 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-73931-1-256 2026-02-25T22:33:31.525 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:33:31.730 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:33:31.731 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:31.731 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:33:31.731 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:33:31.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T22:33:31.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T22:33:31.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2026-02-25T22:33:31.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T22:33:31.992 INFO:tasks.workunit.client.0.trial125.stderr:noout is set 2026-02-25T22:33:32.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2026-02-25T22:33:32.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-73931-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:32.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:33:32.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:33:32.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-25T22:33:32.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:33:32.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-73931-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:32.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:33:32.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:33:32.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-25T22:33:32.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:33:32.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T22:33:32.006 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:33:32.006 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:33:32.006 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:33:32.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:33:32.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:33:32.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:33:32.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-73931-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:32.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:33:32.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:33:32.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-25T22:33:32.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:33:32.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-25T22:33:32.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-73931-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:32.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-25T22:33:32.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:33:32.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:33:32.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T22:33:32.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:33:32.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:33:32.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:33:32.493 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:33:32.494 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:32.495 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:32.495 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:32.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:32.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:33:32.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:33:32.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:33:32.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:33:32.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:33:32.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:33:32.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:33:32.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:33:32.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:33:32.497 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:33:32.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T22:33:32.498 INFO:tasks.workunit.client.0.trial125.stderr:start osd.2 2026-02-25T22:33:32.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:33:32.498 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T22:33:32.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T22:33:32.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:33:32.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:33:32.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:33:32.512 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:32.510+0000 7f8d8a24fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:32.514 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:32.510+0000 7f8d8a24fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:32.515 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:32.510+0000 7f8d8a24fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:32.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T22:33:32.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:33:32.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:33:32.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:33:32.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:33:32.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:32.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:33:32.711 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:33:32.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:32.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:33:32.869 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:32.866+0000 7f8d8a24fa40 -1 Falling back to public interface 2026-02-25T22:33:32.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:33:33.532 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:33.530+0000 7f8d8a24fa40 -1 osd.2 86 log_to_monitors true 2026-02-25T22:33:33.926 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:33:33.926 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:33.926 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:33:33.927 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:33.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:33.927 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:33:34.137 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 up in weight 1 up_from 92 up_thru 80 down_at 89 last_clean_interval [69,86) [v2:127.0.0.1:6818/1388849708,v1:127.0.0.1:6819/1388849708] [v2:127.0.0.1:6820/1388849708,v1:127.0.0.1:6821/1388849708] exists,up 95a7cf33-08b0-44c2-a70d-103d90060fdc 2026-02-25T22:33:34.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:33:34.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:33:34.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:33:34.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:33:34.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:33:34.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:33:34.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:33:34.139 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:33:34.139 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:33:34.139 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:33:34.139 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:33:34.139 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:33:34.139 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:33:34.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:33:34.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:33:34.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:33:34.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:33:34.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:33:34.243 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:33:34.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:33:34.458 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:34.459 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:33:34.459 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:33:34.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:33:34.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:34.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:33:34.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803804 2026-02-25T22:33:34.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803804 2026-02-25T22:33:34.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804' 2026-02-25T22:33:34.537 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:34.537 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:33:34.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285574 2026-02-25T22:33:34.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285574 2026-02-25T22:33:34.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804 1-356482285574' 2026-02-25T22:33:34.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:34.616 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:33:34.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991235 2026-02-25T22:33:34.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991235 2026-02-25T22:33:34.693 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804 1-356482285574 2-395136991235' 2026-02-25T22:33:34.693 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:34.693 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:33:34.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117018 2026-02-25T22:33:34.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117018 2026-02-25T22:33:34.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804 1-356482285574 2-395136991235 3-115964117018' 2026-02-25T22:33:34.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:34.773 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803804 2026-02-25T22:33:34.773 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:34.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:33:34.775 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803804 2026-02-25T22:33:34.775 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:34.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803804 2026-02-25T22:33:34.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803804' 2026-02-25T22:33:34.777 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803804 2026-02-25T22:33:34.777 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:35.000 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803804 -lt 25769803804 2026-02-25T22:33:35.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:35.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-356482285574 2026-02-25T22:33:35.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:35.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:33:35.003 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-356482285574 2026-02-25T22:33:35.004 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:35.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=356482285574 2026-02-25T22:33:35.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 356482285574' 2026-02-25T22:33:35.005 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 356482285574 2026-02-25T22:33:35.005 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:33:35.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 356482285574 -lt 356482285574 2026-02-25T22:33:35.223 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:35.224 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-395136991235 2026-02-25T22:33:35.224 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:35.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:33:35.226 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-395136991235 2026-02-25T22:33:35.226 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:35.227 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=395136991235 2026-02-25T22:33:35.228 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 395136991235' 2026-02-25T22:33:35.228 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 395136991235 2026-02-25T22:33:35.228 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:33:35.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 1 -lt 395136991235 2026-02-25T22:33:35.444 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:33:36.445 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:33:36.445 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:33:36.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 1 -lt 395136991235 2026-02-25T22:33:36.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:33:37.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:33:37.658 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:33:37.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 395136991235 -lt 395136991235 2026-02-25T22:33:37.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:37.870 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117018 2026-02-25T22:33:37.871 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:37.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:33:37.873 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117018 2026-02-25T22:33:37.873 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:37.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117018 2026-02-25T22:33:37.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117018' 2026-02-25T22:33:37.875 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 115964117018 2026-02-25T22:33:37.875 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:33:38.082 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117019 -lt 115964117018 2026-02-25T22:33:38.083 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:33:38.084 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:38.084 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:38.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:33:38.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:33:38.361 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:33:38.361 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:33:38.361 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:33:38.361 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:33:38.361 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:33:38.361 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:33:38.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:33:38.563 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:33:38.563 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:38.563 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:38.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:33:38.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:33:38.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:33:38.835 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-25T22:33:38.837 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records in 2026-02-25T22:33:38.837 INFO:tasks.workunit.client.0.trial125.stderr:1+0 records out 2026-02-25T22:33:38.837 INFO:tasks.workunit.client.0.trial125.stderr:256 bytes copied, 2.4456e-05 s, 10.5 MB/s 2026-02-25T22:33:38.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-73931-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-73931-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:33:38.838 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:33:38.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:33:38.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:33:38.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:33:39.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:33:39.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-73931-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:39.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:33:39.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:33:39.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-25T22:33:39.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:33:39.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-25T22:33:39.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-73931-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T22:33:39.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-25T22:33:39.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:33:39.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:33:39.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:33:39.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:33:39.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:33:39.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:33:39.419 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:33:39.419 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:39.419 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:39.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:39.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:39.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:33:39.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:33:39.421 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr:start osd.2 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T22:33:39.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T22:33:39.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:33:39.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:33:39.423 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:33:39.436 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:39.434+0000 7fa5f8501a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:39.438 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:39.434+0000 7fa5f8501a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:39.439 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:39.434+0000 7fa5f8501a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T22:33:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:33:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:33:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:33:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:33:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:39.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:33:39.631 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:33:39.631 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:39.631 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:33:39.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:33:39.909 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:39.906+0000 7fa5f8501a40 -1 Falling back to public interface 2026-02-25T22:33:40.607 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:40.602+0000 7fa5f8501a40 -1 osd.2 93 log_to_monitors true 2026-02-25T22:33:40.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:33:40.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:40.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:33:40.839 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:40.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:40.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:33:41.046 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 up in weight 1 up_from 97 up_thru 80 down_at 94 last_clean_interval [92,93) [v2:127.0.0.1:6818/2379456323,v1:127.0.0.1:6819/2379456323] [v2:127.0.0.1:6820/2379456323,v1:127.0.0.1:6821/2379456323] exists,up 95a7cf33-08b0-44c2-a70d-103d90060fdc 2026-02-25T22:33:41.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:33:41.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:33:41.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:33:41.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:33:41.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:33:41.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:33:41.047 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:33:41.048 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:33:41.048 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:33:41.048 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:33:41.048 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:33:41.048 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:33:41.048 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:33:41.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:33:41.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:33:41.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:33:41.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:33:41.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:33:41.162 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:33:41.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:33:41.375 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:33:41.375 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:33:41.375 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:33:41.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:33:41.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:41.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:33:41.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803808 2026-02-25T22:33:41.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803808 2026-02-25T22:33:41.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803808' 2026-02-25T22:33:41.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:41.458 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:33:41.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285578 2026-02-25T22:33:41.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285578 2026-02-25T22:33:41.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803808 1-356482285578' 2026-02-25T22:33:41.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:41.536 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:33:41.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=416611827715 2026-02-25T22:33:41.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 416611827715 2026-02-25T22:33:41.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803808 1-356482285578 2-416611827715' 2026-02-25T22:33:41.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:33:41.617 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:33:41.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117022 2026-02-25T22:33:41.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117022 2026-02-25T22:33:41.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803808 1-356482285578 2-416611827715 3-115964117022' 2026-02-25T22:33:41.696 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:41.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803808 2026-02-25T22:33:41.697 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:41.698 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:33:41.699 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803808 2026-02-25T22:33:41.699 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:41.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803808 2026-02-25T22:33:41.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2026-02-25T22:33:41.700 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803808 2026-02-25T22:33:41.700 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:41.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803806 -lt 25769803808 2026-02-25T22:33:41.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:33:42.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:33:42.914 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:33:43.125 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803808 -lt 25769803808 2026-02-25T22:33:43.126 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:43.126 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-356482285578 2026-02-25T22:33:43.127 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:43.128 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:33:43.129 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-356482285578 2026-02-25T22:33:43.129 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:43.130 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=356482285578 2026-02-25T22:33:43.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 356482285578' 2026-02-25T22:33:43.131 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 356482285578 2026-02-25T22:33:43.131 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:33:43.354 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 356482285578 -lt 356482285578 2026-02-25T22:33:43.354 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:43.355 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-416611827715 2026-02-25T22:33:43.355 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:43.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:33:43.357 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-416611827715 2026-02-25T22:33:43.358 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:43.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=416611827715 2026-02-25T22:33:43.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 416611827715' 2026-02-25T22:33:43.359 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 416611827715 2026-02-25T22:33:43.359 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:33:43.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 416611827715 -lt 416611827715 2026-02-25T22:33:43.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:33:43.575 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117022 2026-02-25T22:33:43.575 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:33:43.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:33:43.577 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117022 2026-02-25T22:33:43.577 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:33:43.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117022 2026-02-25T22:33:43.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117022' 2026-02-25T22:33:43.579 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 115964117022 2026-02-25T22:33:43.579 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:33:43.802 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117022 -lt 115964117022 2026-02-25T22:33:43.803 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:33:43.804 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:43.804 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:44.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:33:44.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:33:44.089 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:33:44.089 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:33:44.089 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:33:44.089 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:33:44.089 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:33:44.089 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:33:44.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-25T22:33:44.297 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:33:44.297 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:44.297 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:44.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-25T22:33:44.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' -1 2026-02-25T22:33:44.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-25T22:33:44.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=3 2026-02-25T22:33:44.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:33:44.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-25T22:33:44.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:33:44.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:33:44.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:33:44.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:33:44.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:33:44.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:33:44.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:33:44.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:33:44.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-25T22:33:44.887 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:33:44.888 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:44.888 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:45.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-25T22:33:45.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-25T22:33:45.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:33:45.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:33:45.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:33:45.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:33:45.173 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:33:45.173 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:33:45.174 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:33:45.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:33:45.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:33:45.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-25T22:33:45.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:33:45.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-25T22:33:45.661 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:33:45.661 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:33:45.662 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:33:45.662 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:33:45.662 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:33:45.662 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:33:45.663 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:33:45.663 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:33:45.872 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-25T22:33:45.873 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:33:45.873 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:45.873 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:46.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-25T22:33:46.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-25T22:33:46.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:33:46.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:33:46.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:33:46.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:33:46.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:33:46.158 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:33:46.158 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:33:46.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:33:46.441 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:33:46.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-25T22:33:46.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:33:46.442 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-25T22:33:46.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:33:46.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:33:46.843 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:33:46.843 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:33:46.843 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:33:46.843 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:33:46.844 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:33:46.844 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:33:47.055 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:33:47.056 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:33:47.056 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:33:47.056 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:33:47.346 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:33:47.346 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:33:47.346 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:33:47.346 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T22:33:47.347 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T22:33:47.700 INFO:tasks.workunit.client.0.trial125.stderr:noout is unset 2026-02-25T22:33:47.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2026-02-25T22:33:47.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-73931-1-256 2026-02-25T22:33:47.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:33:47.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:33:47.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-73931-1-256 2026-02-25T22:33:47.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:33:47.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:33:47.713 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-73931-1-256 td/test-erasure-eio/COPY 2026-02-25T22:33:47.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:33:47.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:33:47.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T22:33:47.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:346: TEST_rados_get_bad_size_shard_1: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:33:47.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:33:47.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:33:47.953 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:33:47.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:33:48.237 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:33:48.248 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:33:48.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:33:48.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:33:48.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:33:48.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:33:48.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:33:48.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:33:48.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:33:48.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:33:48.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:33:48.360 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:33:48.361 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:33:48.362 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:33:48.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:33:48.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:33:48.364 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:33:48.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:33:48.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:33:48.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:33:48.365 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:33:48.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:33:48.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:33:48.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:33:48.390 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:33:48.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:48.391 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:48.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:33:48.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:33:48.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:33:48.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:33:48.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:33:48.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:33:48.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:33:48.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:33:48.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:33:48.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:33:48.394 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:33:48.394 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:33:48.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:33:48.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:33:48.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:33:48.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:33:48.396 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:33:48.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:33:48.397 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:33:48.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:33:48.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:33:48.399 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:33:48.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:33:48.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:33:48.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:33:48.401 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:33:48.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:33:48.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:33:48.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:33:48.404 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:33:48.405 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:48.405 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:48.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:33:48.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:33:48.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:33:48.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:33:48.408 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:33:48.408 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:48.408 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:48.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:33:48.410 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:33:48.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:33:48.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:33:48.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:33:48.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:33:48.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:33:48.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:33:48.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:33:48.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:33:48.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:33:48.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:33:48.432 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:33:48.432 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:33:48.432 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:33:48.433 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:33:48.433 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:48.433 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:48.433 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:48.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:33:48.463 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:33:48.464 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:33:48.465 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:33:48.465 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:33:48.465 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:33:48.465 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:33:48.466 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:33:48.466 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:33:48.466 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:33:48.466 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:33:48.466 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:48.466 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:48.466 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:33:48.467 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:33:48.467 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:33:48.521 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:33:48.521 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:33:48.522 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:33:48.522 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:33:48.523 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:33:48.523 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:33:48.523 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:33:48.523 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:33:48.523 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:33:48.523 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:48.523 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:48.523 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:33:48.524 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:33:48.524 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:33:48.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:33:48.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:33:48.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:33:48.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:33:48.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:33:48.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:33:48.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:33:48.683 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:33:48.683 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:33:48.683 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:33:48.684 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:33:48.684 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:48.684 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:48.684 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:48.684 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:33:48.686 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:33:48.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:33:48.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:33:48.810 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:33:48.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:33:49.822 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:33:49.822 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:33:49.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:33:49.823 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:33:49.823 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:49.823 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:49.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:33:49.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:33:49.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:33:49.871 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:33:49.877 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:33:49.878 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:33:48.450+0000 7f62d79dbec0 0 load: jerasure load: lrc 2026-02-25T22:33:49.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_0 td/test-erasure-eio 2026-02-25T22:33:49.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:275: TEST_rados_get_subread_eio_shard_0: local dir=td/test-erasure-eio 2026-02-25T22:33:49.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:276: TEST_rados_get_subread_eio_shard_0: setup_osds 4 2026-02-25T22:33:49.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T22:33:49.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:33:49.880 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T22:33:49.881 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T22:33:49.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:33:49.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:33:49.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:33:49.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:33:49.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:33:49.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:33:49.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:33:49.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:33:49.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:33:49.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:33:49.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:33:49.886 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:33:49.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:33:49.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:33:49.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:33:49.887 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:33:49.887 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:33:49.887 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:33:49.887 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:33:49.887 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:33:49.887 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:49.887 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:49.888 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:49.888 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:49.888 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:33:49.888 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:33:49.888 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:33:49.888 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:33:49.888 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:33:49.888 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:33:49.888 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:33:49.888 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:33:49.889 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:33:49.889 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:33:49.889 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:33:49.889 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:33:49.889 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:33:49.889 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=23cdbdb1-c221-4967-8c58-016b433c7313 2026-02-25T22:33:49.889 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 23cdbdb1-c221-4967-8c58-016b433c7313 2026-02-25T22:33:49.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 23cdbdb1-c221-4967-8c58-016b433c7313' 2026-02-25T22:33:49.890 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:33:49.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDNeJ9pWG5cNRAAb/AvvNJUpTOOw7AaZjQC4g== 2026-02-25T22:33:49.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDNeJ9pWG5cNRAAb/AvvNJUpTOOw7AaZjQC4g=="}' 2026-02-25T22:33:49.897 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 23cdbdb1-c221-4967-8c58-016b433c7313 -i td/test-erasure-eio/0/new.json 2026-02-25T22:33:50.006 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:33:50.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:33:50.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNeJ9pWG5cNRAAb/AvvNJUpTOOw7AaZjQC4g== --osd-uuid 23cdbdb1-c221-4967-8c58-016b433c7313 2026-02-25T22:33:50.029 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:50.026+0000 7f9e4def9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:50.031 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:50.026+0000 7f9e4def9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:50.032 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:50.030+0000 7f9e4def9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:50.033 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:50.030+0000 7f9e4def9a40 -1 bdev(0x55aa4f7c2400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:33:50.033 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:50.030+0000 7f9e4def9a40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:33:50.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:33:50.811 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:33:50.812 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:33:50.813 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:33:50.813 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:33:50.951 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:33:50.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:33:50.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:33:50.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:33:50.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:33:50.953 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:33:50.976 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:50.974+0000 7f80583f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:50.978 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:50.974+0000 7f80583f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:50.979 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:50.974+0000 7f80583f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:51.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:33:51.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:33:51.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:33:51.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:33:51.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:33:51.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:51.058 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:33:51.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:33:51.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:51.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:33:51.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:33:51.445 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:51.442+0000 7f80583f0a40 -1 Falling back to public interface 2026-02-25T22:33:52.081 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:52.078+0000 7f80583f0a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:33:52.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:33:52.271 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:52.271 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:33:52.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:33:52.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:52.272 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:33:52.484 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:33:53.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:33:53.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:53.486 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:33:53.487 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:33:53.487 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:53.487 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:33:53.699 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1067037812,v1:127.0.0.1:6803/1067037812] [v2:127.0.0.1:6804/1067037812,v1:127.0.0.1:6805/1067037812] exists,up 23cdbdb1-c221-4967-8c58-016b433c7313 2026-02-25T22:33:53.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:33:53.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:33:53.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:33:53.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:33:53.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:33:53.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:33:53.704 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:33:53.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:33:53.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:33:53.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:33:53.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:33:53.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:33:53.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:33:53.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:33:53.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:33:53.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:33:53.707 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:33:53.707 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=010d14e7-19ab-4c64-a902-e17266edd368 2026-02-25T22:33:53.707 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 010d14e7-19ab-4c64-a902-e17266edd368 2026-02-25T22:33:53.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 010d14e7-19ab-4c64-a902-e17266edd368' 2026-02-25T22:33:53.708 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:33:53.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDReJ9ppRCaKhAAHoLaOIqeMUHGqkETwVn4pQ== 2026-02-25T22:33:53.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDReJ9ppRCaKhAAHoLaOIqeMUHGqkETwVn4pQ=="}' 2026-02-25T22:33:53.716 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 010d14e7-19ab-4c64-a902-e17266edd368 -i td/test-erasure-eio/1/new.json 2026-02-25T22:33:53.929 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:33:53.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:33:53.939 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDReJ9ppRCaKhAAHoLaOIqeMUHGqkETwVn4pQ== --osd-uuid 010d14e7-19ab-4c64-a902-e17266edd368 2026-02-25T22:33:53.956 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:53.954+0000 7f3b0e04aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:53.958 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:53.954+0000 7f3b0e04aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:53.959 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:53.954+0000 7f3b0e04aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:53.959 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:53.954+0000 7f3b0e04aa40 -1 bdev(0x56464ad66400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:33:53.959 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:53.958+0000 7f3b0e04aa40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:33:54.574 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:33:54.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:33:54.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:33:54.576 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:33:54.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:33:54.869 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:33:54.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:33:54.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:33:54.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:33:54.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:33:54.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:33:54.892 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:54.890+0000 7ff768943a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:54.894 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:54.890+0000 7ff768943a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:54.895 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:54.890+0000 7ff768943a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:55.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:33:55.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:33:55.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:33:55.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:33:55.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:33:55.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:55.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:33:55.087 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:33:55.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:55.088 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:33:55.299 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:33:55.373 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:55.370+0000 7ff768943a40 -1 Falling back to public interface 2026-02-25T22:33:56.053 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:56.050+0000 7ff768943a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:33:56.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:33:56.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:56.300 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:33:56.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:33:56.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:56.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:33:56.513 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:33:57.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:33:57.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:57.515 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:33:57.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:33:57.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:57.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:33:57.731 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3586842766,v1:127.0.0.1:6811/3586842766] [v2:127.0.0.1:6812/3586842766,v1:127.0.0.1:6813/3586842766] exists,up 010d14e7-19ab-4c64-a902-e17266edd368 2026-02-25T22:33:57.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:33:57.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:33:57.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:33:57.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:33:57.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:33:57.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:33:57.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:33:57.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:33:57.732 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:33:57.733 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:33:57.735 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:33:57.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=04293d7b-501a-41da-b268-cf7f4fbe3682 2026-02-25T22:33:57.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 04293d7b-501a-41da-b268-cf7f4fbe3682' 2026-02-25T22:33:57.735 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 04293d7b-501a-41da-b268-cf7f4fbe3682 2026-02-25T22:33:57.736 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:33:57.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDVeJ9p5It/LBAARNiuYdGL/z5b3zIX9wsftQ== 2026-02-25T22:33:57.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDVeJ9p5It/LBAARNiuYdGL/z5b3zIX9wsftQ=="}' 2026-02-25T22:33:57.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 04293d7b-501a-41da-b268-cf7f4fbe3682 -i td/test-erasure-eio/2/new.json 2026-02-25T22:33:57.962 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:33:57.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:33:57.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVeJ9p5It/LBAARNiuYdGL/z5b3zIX9wsftQ== --osd-uuid 04293d7b-501a-41da-b268-cf7f4fbe3682 2026-02-25T22:33:57.989 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:57.986+0000 7f856c4b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:57.990 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:57.986+0000 7f856c4b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:57.991 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:57.990+0000 7f856c4b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:57.991 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:57.990+0000 7f856c4b1a40 -1 bdev(0x564875de8400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:33:57.992 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:57.990+0000 7f856c4b1a40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:33:58.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:33:58.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:33:58.640 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:33:58.640 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:33:58.640 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:33:58.937 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:33:58.938 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:33:58.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:33:58.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:33:58.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:33:58.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:33:58.961 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:58.958+0000 7fb00cc60a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:58.963 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:58.958+0000 7fb00cc60a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:58.964 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:58.962+0000 7fb00cc60a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:33:59.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:33:59.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:33:59.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:33:59.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:33:59.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:33:59.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:33:59.145 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:33:59.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:33:59.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:33:59.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:33:59.281 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:59.278+0000 7fb00cc60a40 -1 Falling back to public interface 2026-02-25T22:33:59.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:33:59.944 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:33:59.941+0000 7fb00cc60a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:34:00.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:00.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:00.359 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:00.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:34:00.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:00.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:34:00.565 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:01.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:01.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:01.567 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:34:01.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:34:01.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:01.567 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:34:01.776 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2644150670,v1:127.0.0.1:6819/2644150670] [v2:127.0.0.1:6820/2644150670,v1:127.0.0.1:6821/2644150670] exists,up 04293d7b-501a-41da-b268-cf7f4fbe3682 2026-02-25T22:34:01.776 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:34:01.776 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:01.777 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:01.778 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:01.778 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:01.778 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:01.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:01.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:34:01.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:34:01.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:34:01.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:34:01.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:34:01.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:34:01.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:34:01.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:34:01.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:34:01.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:34:01.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:34:01.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:34:01.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:34:01.780 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1873b01c-f29e-4213-b994-da672d9f2827 2026-02-25T22:34:01.780 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 1873b01c-f29e-4213-b994-da672d9f2827 2026-02-25T22:34:01.780 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 1873b01c-f29e-4213-b994-da672d9f2827' 2026-02-25T22:34:01.781 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:34:01.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDZeJ9p+LdCLxAAyR14AnO9TZnoIVLo+JgjHA== 2026-02-25T22:34:01.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDZeJ9p+LdCLxAAyR14AnO9TZnoIVLo+JgjHA=="}' 2026-02-25T22:34:01.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1873b01c-f29e-4213-b994-da672d9f2827 -i td/test-erasure-eio/3/new.json 2026-02-25T22:34:02.010 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:34:02.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:34:02.020 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZeJ9p+LdCLxAAyR14AnO9TZnoIVLo+JgjHA== --osd-uuid 1873b01c-f29e-4213-b994-da672d9f2827 2026-02-25T22:34:02.044 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:02.041+0000 7f80b3703a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:02.046 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:02.041+0000 7f80b3703a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:02.046 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:02.041+0000 7f80b3703a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:02.047 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:02.041+0000 7f80b3703a40 -1 bdev(0x560013cd6400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:34:02.047 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:02.045+0000 7f80b3703a40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:34:02.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:34:02.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:34:02.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:34:02.684 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:34:02.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:34:02.977 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:34:02.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:34:02.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:34:02.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:34:02.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:34:02.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:34:02.993 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:02.989+0000 7facc27b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:02.995 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:02.989+0000 7facc27b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:02.996 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:02.993+0000 7facc27b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:03.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:34:03.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:34:03.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:34:03.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:34:03.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:34:03.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:03.190 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:34:03.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:34:03.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:03.191 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:34:03.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:03.465 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:03.461+0000 7facc27b7a40 -1 Falling back to public interface 2026-02-25T22:34:04.045 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:04.041+0000 7facc27b7a40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:34:04.404 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:04.404 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:04.404 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:34:04.405 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:04.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:04.405 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:34:04.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:05.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:05.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:05.614 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:34:05.615 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:34:05.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:05.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:34:05.826 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 26 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4181470447,v1:127.0.0.1:6827/4181470447] [v2:127.0.0.1:6828/4181470447,v1:127.0.0.1:6829/4181470447] exists,up 1873b01c-f29e-4213-b994-da672d9f2827 2026-02-25T22:34:05.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:34:05.826 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:34:05.827 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:34:05.827 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:34:05.828 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:34:05.828 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:34:05.828 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:05.828 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:05.828 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:05.828 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:34:05.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:34:05.828 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:34:05.884 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:34:05.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:34:05.892 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:33:51.446+0000 7f80583f0a40 0 load: jerasure load: lrc 2026-02-25T22:34:05.893 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:278: TEST_rados_get_subread_eio_shard_0: local poolname=pool-jerasure 2026-02-25T22:34:05.893 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:279: TEST_rados_get_subread_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T22:34:05.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:34:05.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:34:05.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T22:34:05.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:34:05.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T22:34:05.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:34:05.894 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T22:34:06.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:34:06.168 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:34:06.533 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:34:06.543 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:34:07.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:34:07.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:34:07.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:34:07.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:34:07.546 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:34:07.546 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:34:07.546 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:34:07.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:34:07.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:34:07.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:34:07.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:34:07.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:34:07.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:34:07.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:34:07.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:34:07.658 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:34:07.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:34:07.865 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:34:07.865 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:34:07.865 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:34:07.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:34:07.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:07.866 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:34:07.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803782 2026-02-25T22:34:07.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803782 2026-02-25T22:34:07.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782' 2026-02-25T22:34:07.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:07.945 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:34:08.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574853 2026-02-25T22:34:08.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574853 2026-02-25T22:34:08.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574853' 2026-02-25T22:34:08.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:08.024 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:34:08.102 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378628 2026-02-25T22:34:08.102 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378628 2026-02-25T22:34:08.102 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574853 2-81604378628' 2026-02-25T22:34:08.102 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:08.102 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:34:08.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149700 2026-02-25T22:34:08.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149700 2026-02-25T22:34:08.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574853 2-81604378628 3-111669149700' 2026-02-25T22:34:08.180 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:08.180 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803782 2026-02-25T22:34:08.181 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:08.181 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:34:08.182 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803782 2026-02-25T22:34:08.182 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:08.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803782 2026-02-25T22:34:08.184 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803782' 2026-02-25T22:34:08.184 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803782 2026-02-25T22:34:08.184 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:34:08.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803780 -lt 25769803782 2026-02-25T22:34:08.394 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:34:09.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:34:09.396 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:34:09.603 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803783 -lt 25769803782 2026-02-25T22:34:09.603 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:09.603 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574853 2026-02-25T22:34:09.604 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:09.605 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:34:09.606 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574853 2026-02-25T22:34:09.606 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:09.607 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574853 2026-02-25T22:34:09.608 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574853 2026-02-25T22:34:09.608 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574853' 2026-02-25T22:34:09.608 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:34:09.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574853 -lt 55834574853 2026-02-25T22:34:09.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:09.819 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378628 2026-02-25T22:34:09.819 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:09.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:34:09.821 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378628 2026-02-25T22:34:09.821 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:09.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378628 2026-02-25T22:34:09.823 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 81604378628 2026-02-25T22:34:09.823 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378628' 2026-02-25T22:34:09.823 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:34:10.044 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378628 -lt 81604378628 2026-02-25T22:34:10.044 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:10.045 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149700 2026-02-25T22:34:10.045 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:10.046 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:34:10.047 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149700 2026-02-25T22:34:10.047 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:10.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149700 2026-02-25T22:34:10.048 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149700' 2026-02-25T22:34:10.049 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149700 2026-02-25T22:34:10.049 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:34:10.253 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149700 -lt 111669149700 2026-02-25T22:34:10.253 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:34:10.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:34:10.254 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:34:10.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:34:10.532 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:34:10.533 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:34:10.533 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:34:10.533 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:34:10.534 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:34:10.534 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:34:10.534 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:34:10.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:34:10.739 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:34:10.739 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:34:10.739 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:34:11.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:34:11.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:34:11.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:34:11.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:281: TEST_rados_get_subread_eio_shard_0: local shard_id=0 2026-02-25T22:34:11.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:282: TEST_rados_get_subread_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 2026-02-25T22:34:11.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-25T22:34:11.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-25T22:34:11.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-25T22:34:11.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-25T22:34:11.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2026-02-25T22:34:11.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-73931-0 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-73931-0 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-73931-0 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:34:11.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:11.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:34:11.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:11.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:34:11.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-73931-0 td/test-erasure-eio/ORIGINAL 2026-02-25T22:34:11.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-73931-0 td/test-erasure-eio 0 2026-02-25T22:34:11.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:34:11.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-73931-0 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-73931-0 2026-02-25T22:34:11.036 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:34:11.037 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-73931-0 2026-02-25T22:34:11.037 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-73931-0 2026-02-25T22:34:11.037 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:34:11.248 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:34:11.249 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:34:11.249 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:34:11.249 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:34:11.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T22:34:11.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:34:11.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T22:34:11.249 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:34:11.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T22:34:11.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:34:11.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T22:34:11.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:34:11.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T22:34:11.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:34:11.251 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:34:11.252 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:34:11.252 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T22:34:11.252 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:34:11.253 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:34:11.253 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:11.253 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:11.253 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:11.253 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:34:11.253 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:34:11.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:34:11.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:34:11.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:34:11.315 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T22:34:11.315 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:34:11.315 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:34:11.316 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:11.316 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:11.316 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:11.316 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:34:11.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:34:11.317 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-73931-0 0 2026-02-25T22:34:11.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-73931-0 2026-02-25T22:34:11.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:34:11.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:34:11.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-73931-0 2026-02-25T22:34:11.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:34:11.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:34:11.377 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-73931-0 td/test-erasure-eio/COPY 2026-02-25T22:34:11.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:34:11.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:34:11.398 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2026-02-25T22:34:11.399 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 0 + 1 2026-02-25T22:34:11.401 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=1 2026-02-25T22:34:11.401 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-73931-0 td/test-erasure-eio 1 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-73931-0 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T22:34:11.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:34:11.403 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-73931-0 2026-02-25T22:34:11.403 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:34:11.403 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-73931-0 2026-02-25T22:34:11.403 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-73931-0 2026-02-25T22:34:11.403 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:34:11.614 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:34:11.614 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:34:11.614 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:34:11.614 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:34:11.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T22:34:11.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:34:11.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=1 2026-02-25T22:34:11.615 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:34:11.615 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/1/type 2026-02-25T22:34:11.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:34:11.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-25T22:34:11.616 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:34:11.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-25T22:34:11.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:34:11.617 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:34:11.617 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:34:11.617 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-25T22:34:11.617 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:34:11.617 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:34:11.618 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:11.618 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:11.618 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:11.618 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.1.asok 2026-02-25T22:34:11.618 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:34:11.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:34:11.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:34:11.678 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:34:11.679 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.1 2026-02-25T22:34:11.679 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:34:11.679 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:34:11.680 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:11.680 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:11.680 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:11.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.1.asok 2026-02-25T22:34:11.680 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:34:11.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-73931-0 1 2026-02-25T22:34:11.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 0 -eq 1 -a 1 -eq 2 ']' 2026-02-25T22:34:11.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-73931-0 fail 2026-02-25T22:34:11.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:34:11.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:34:11.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-73931-0 2026-02-25T22:34:11.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-25T22:34:11.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-25T22:34:11.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-73931-0 td/test-erasure-eio/COPY 2026-02-25T22:34:11.759 INFO:tasks.workunit.client.0.trial125.stderr:error getting pool-jerasure/obj-eio-73931-0: (5) Input/output error 2026-02-25T22:34:11.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-25T22:34:11.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2026-02-25T22:34:11.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:283: TEST_rados_get_subread_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:34:11.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:34:11.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:34:12.024 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:34:12.035 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:34:12.295 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:34:12.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:34:12.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:34:12.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:34:12.306 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:34:12.307 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:34:12.307 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:34:12.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:34:12.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:34:12.307 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:34:12.619 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:34:12.619 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:34:12.620 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:34:12.621 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:34:12.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:34:12.623 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:34:12.623 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:34:12.624 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:34:12.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:34:12.625 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:34:12.625 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:34:12.626 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:34:12.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:34:12.628 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:34:12.639 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:34:12.639 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:12.639 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:12.640 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:34:12.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:34:12.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:34:12.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:34:12.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:34:12.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:34:12.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:34:12.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:34:12.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:34:12.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:34:12.643 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:34:12.643 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:34:12.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:34:12.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:34:12.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:34:12.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:34:12.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:34:12.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:34:12.646 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:34:12.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:34:12.647 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:34:12.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:34:12.648 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:34:12.648 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:34:12.648 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:34:12.649 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:34:12.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:34:12.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:34:12.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:34:12.653 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:34:12.653 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:12.653 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:12.653 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:34:12.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:34:12.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:34:12.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:34:12.656 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:34:12.656 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:12.656 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:12.656 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:34:12.658 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:34:12.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:34:12.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:34:12.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:34:12.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:34:12.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:34:12.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:34:12.660 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:34:12.660 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:34:12.660 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:34:12.660 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:34:12.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:34:12.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:12.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:12.681 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:12.681 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:12.681 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:12.681 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:12.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:34:12.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:34:12.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:34:12.719 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:34:12.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:34:12.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:34:12.721 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:34:12.721 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:34:12.721 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:34:12.721 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:34:12.721 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:12.721 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:12.721 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:12.721 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:34:12.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:34:12.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:34:12.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:34:12.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:34:12.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:34:12.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:34:12.780 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:34:12.780 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:34:12.780 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:34:12.780 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:34:12.780 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:12.781 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:12.781 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:12.781 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:34:12.781 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:34:12.781 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:34:12.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:34:12.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:34:12.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:34:12.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:34:12.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:34:12.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:34:12.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:34:12.948 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:34:12.948 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:12.949 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:12.949 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:12.949 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:12.950 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:12.950 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:12.950 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:34:12.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:34:12.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:34:12.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:34:13.088 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:34:13.096 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:34:14.098 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:34:14.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:34:14.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:34:14.099 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:14.099 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:14.099 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:14.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:34:14.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:34:14.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:34:14.148 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:34:14.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:34:14.155 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:34:12.701+0000 7fc550bfdec0 0 load: jerasure load: lrc 2026-02-25T22:34:14.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_1 td/test-erasure-eio 2026-02-25T22:34:14.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:287: TEST_rados_get_subread_eio_shard_1: local dir=td/test-erasure-eio 2026-02-25T22:34:14.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:288: TEST_rados_get_subread_eio_shard_1: setup_osds 4 2026-02-25T22:34:14.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T22:34:14.157 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:34:14.157 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T22:34:14.158 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T22:34:14.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:34:14.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:34:14.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:34:14.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:34:14.161 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:34:14.162 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:34:14.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:34:14.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:34:14.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:34:14.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:34:14.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:34:14.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:34:14.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:34:14.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:34:14.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:34:14.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=731db30a-c5bf-4cb8-9200-9d387be2ca6d 2026-02-25T22:34:14.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 731db30a-c5bf-4cb8-9200-9d387be2ca6d' 2026-02-25T22:34:14.165 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 731db30a-c5bf-4cb8-9200-9d387be2ca6d 2026-02-25T22:34:14.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:34:14.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDmeJ9py4JIChAA1N+IhElX7hqsHqGZu67XjQ== 2026-02-25T22:34:14.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDmeJ9py4JIChAA1N+IhElX7hqsHqGZu67XjQ=="}' 2026-02-25T22:34:14.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 731db30a-c5bf-4cb8-9200-9d387be2ca6d -i td/test-erasure-eio/0/new.json 2026-02-25T22:34:14.287 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:34:14.294 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:34:14.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmeJ9py4JIChAA1N+IhElX7hqsHqGZu67XjQ== --osd-uuid 731db30a-c5bf-4cb8-9200-9d387be2ca6d 2026-02-25T22:34:14.312 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:14.309+0000 7f1e3c7e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:14.313 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:14.309+0000 7f1e3c7e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:14.314 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:14.309+0000 7f1e3c7e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:14.315 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:14.309+0000 7f1e3c7e4a40 -1 bdev(0x5630fbc18400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:34:14.315 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:14.309+0000 7f1e3c7e4a40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:34:15.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:34:15.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:34:15.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:34:15.017 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:34:15.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:34:15.156 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:34:15.157 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:34:15.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:34:15.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:34:15.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:34:15.158 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:34:15.172 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:15.169+0000 7f03243dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:15.173 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:15.169+0000 7f03243dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:15.174 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:15.169+0000 7f03243dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:15.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:34:15.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:34:15.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:34:15.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:34:15.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:34:15.264 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:34:15.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:15.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:34:15.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:15.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:34:15.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:15.573 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:15.569+0000 7f03243dea40 -1 Falling back to public interface 2026-02-25T22:34:16.205 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:16.201+0000 7f03243dea40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:34:16.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:16.475 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:16.476 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:16.476 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:34:16.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:16.477 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:34:16.689 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:17.690 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:17.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:17.691 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:34:17.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:34:17.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:17.692 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:34:17.899 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1915020747,v1:127.0.0.1:6803/1915020747] [v2:127.0.0.1:6804/1915020747,v1:127.0.0.1:6805/1915020747] exists,up 731db30a-c5bf-4cb8-9200-9d387be2ca6d 2026-02-25T22:34:17.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:34:17.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:34:17.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:34:17.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:34:17.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:34:17.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:34:17.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:34:17.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:34:17.904 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:34:17.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:34:17.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:34:17.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:34:17.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:34:17.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:34:17.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:34:17.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:34:17.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:34:17.906 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:34:17.907 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=62fa668a-0b3b-4880-8435-6444e2f4c0df 2026-02-25T22:34:17.907 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 62fa668a-0b3b-4880-8435-6444e2f4c0df 2026-02-25T22:34:17.908 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 62fa668a-0b3b-4880-8435-6444e2f4c0df' 2026-02-25T22:34:17.908 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:34:17.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDpeJ9ptPqONhAAgdBciSiXZAXNvLQvxR4BUQ== 2026-02-25T22:34:17.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDpeJ9ptPqONhAAgdBciSiXZAXNvLQvxR4BUQ=="}' 2026-02-25T22:34:17.917 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 62fa668a-0b3b-4880-8435-6444e2f4c0df -i td/test-erasure-eio/1/new.json 2026-02-25T22:34:18.136 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:18.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:34:18.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpeJ9ptPqONhAAgdBciSiXZAXNvLQvxR4BUQ== --osd-uuid 62fa668a-0b3b-4880-8435-6444e2f4c0df 2026-02-25T22:34:18.172 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:18.169+0000 7f166e5eea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:18.173 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:18.169+0000 7f166e5eea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:18.174 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:18.169+0000 7f166e5eea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:18.175 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:18.169+0000 7f166e5eea40 -1 bdev(0x55670f65a400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:34:18.175 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:18.173+0000 7f166e5eea40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:34:18.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:34:18.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:34:18.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:34:18.836 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:34:18.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:34:19.120 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:34:19.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:34:19.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:34:19.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:34:19.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:34:19.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:34:19.142 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:19.137+0000 7f9f2e23da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:19.144 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:19.141+0000 7f9f2e23da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:19.146 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:19.141+0000 7f9f2e23da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:19.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:34:19.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:34:19.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:34:19.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:34:19.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:34:19.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:19.332 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:34:19.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:34:19.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:19.332 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:34:19.544 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:19.633 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:19.629+0000 7f9f2e23da40 -1 Falling back to public interface 2026-02-25T22:34:20.250 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:20.245+0000 7f9f2e23da40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:34:20.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:20.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:20.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:34:20.545 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:20.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:20.546 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:34:20.753 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:21.755 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:21.755 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:21.756 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:34:21.757 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:34:21.757 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:21.757 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:34:21.967 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4035952567,v1:127.0.0.1:6811/4035952567] [v2:127.0.0.1:6812/4035952567,v1:127.0.0.1:6813/4035952567] exists,up 62fa668a-0b3b-4880-8435-6444e2f4c0df 2026-02-25T22:34:21.970 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:34:21.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:34:21.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:34:21.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:34:21.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:34:21.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:34:21.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:34:21.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:34:21.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:34:21.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:34:21.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:34:21.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:34:21.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:34:21.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:34:21.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:34:21.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:34:21.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:34:21.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:34:21.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:34:21.973 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:34:21.973 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:21.973 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:21.973 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:21.973 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:21.973 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:21.973 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:21.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:21.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:34:21.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:34:21.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:34:21.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:34:21.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:34:21.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:34:21.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:34:21.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:34:21.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:34:21.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:34:21.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:34:21.974 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:34:21.975 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:34:21.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cff0c3d8-9cd3-42b1-988a-970c55f27c89 2026-02-25T22:34:21.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 cff0c3d8-9cd3-42b1-988a-970c55f27c89' 2026-02-25T22:34:21.976 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 cff0c3d8-9cd3-42b1-988a-970c55f27c89 2026-02-25T22:34:21.976 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:34:21.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDteJ9poq+gOhAA1DQSe6VNc17DGKRiI4+h0Q== 2026-02-25T22:34:21.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDteJ9poq+gOhAA1DQSe6VNc17DGKRiI4+h0Q=="}' 2026-02-25T22:34:21.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cff0c3d8-9cd3-42b1-988a-970c55f27c89 -i td/test-erasure-eio/2/new.json 2026-02-25T22:34:22.200 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:34:22.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:34:22.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDteJ9poq+gOhAA1DQSe6VNc17DGKRiI4+h0Q== --osd-uuid cff0c3d8-9cd3-42b1-988a-970c55f27c89 2026-02-25T22:34:22.225 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:22.221+0000 7fdced442a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:22.227 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:22.221+0000 7fdced442a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:22.228 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:22.225+0000 7fdced442a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:22.228 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:22.225+0000 7fdced442a40 -1 bdev(0x560371e84400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:34:22.228 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:22.225+0000 7fdced442a40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:34:22.855 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:34:22.855 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:34:22.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:34:22.856 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:34:22.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:34:23.141 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:34:23.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:34:23.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:34:23.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:34:23.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:34:23.141 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:34:23.164 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:23.161+0000 7f327f364a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:23.166 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:23.161+0000 7f327f364a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:23.167 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:23.161+0000 7f327f364a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:23.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:34:23.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:34:23.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:34:23.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:34:23.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:34:23.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:23.356 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:34:23.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:34:23.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:23.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:34:23.575 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:23.585 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:23.581+0000 7f327f364a40 -1 Falling back to public interface 2026-02-25T22:34:24.168 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:24.165+0000 7f327f364a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:34:24.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:24.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:24.576 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:34:24.576 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:24.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:24.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:34:24.791 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4114001420,v1:127.0.0.1:6819/4114001420] [v2:127.0.0.1:6820/4114001420,v1:127.0.0.1:6821/4114001420] exists,up cff0c3d8-9cd3-42b1-988a-970c55f27c89 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:34:24.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:34:24.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:34:24.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:34:24.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:34:24.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:34:24.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:34:24.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:24.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:24.793 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:24.793 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:24.793 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:24.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:24.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:24.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:34:24.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:34:24.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:34:24.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:34:24.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:34:24.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:34:24.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:34:24.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:34:24.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:34:24.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:34:24.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:34:24.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:34:24.796 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:34:24.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2209119d-03bd-46eb-a070-528d29935b4e 2026-02-25T22:34:24.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 2209119d-03bd-46eb-a070-528d29935b4e' 2026-02-25T22:34:24.797 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 2209119d-03bd-46eb-a070-528d29935b4e 2026-02-25T22:34:24.797 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:34:24.810 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDweJ9pOew1MBAAmWWVudmKdOyuTmtbCHkeAA== 2026-02-25T22:34:24.810 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDweJ9pOew1MBAAmWWVudmKdOyuTmtbCHkeAA=="}' 2026-02-25T22:34:24.810 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2209119d-03bd-46eb-a070-528d29935b4e -i td/test-erasure-eio/3/new.json 2026-02-25T22:34:25.026 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:34:25.036 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:34:25.037 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDweJ9pOew1MBAAmWWVudmKdOyuTmtbCHkeAA== --osd-uuid 2209119d-03bd-46eb-a070-528d29935b4e 2026-02-25T22:34:25.053 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:25.049+0000 7f7fd6456a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:25.055 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:25.049+0000 7f7fd6456a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:25.056 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:25.053+0000 7f7fd6456a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:25.056 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:25.053+0000 7f7fd6456a40 -1 bdev(0x55f57f4c4400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:34:25.057 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:25.053+0000 7f7fd6456a40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:34:25.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:34:25.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:34:25.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:34:25.711 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:34:25.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:34:25.994 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:34:25.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:34:25.994 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:34:25.995 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:34:25.995 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:34:25.995 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:34:26.017 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:26.013+0000 7f6b66458a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:26.019 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:26.017+0000 7f6b66458a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:26.020 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:26.017+0000 7f6b66458a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:26.206 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:34:26.206 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:34:26.206 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:34:26.206 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:34:26.206 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:34:26.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:26.207 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:34:26.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:34:26.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:26.207 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:34:26.381 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:26.377+0000 7f6b66458a40 -1 Falling back to public interface 2026-02-25T22:34:26.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:27.020 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:27.017+0000 7f6b66458a40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:34:27.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:27.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:27.420 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:27.420 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:34:27.420 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:27.420 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:34:27.642 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:28.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:28.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:28.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:34:28.644 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:34:28.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:28.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:34:28.855 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/445214574,v1:127.0.0.1:6827/445214574] [v2:127.0.0.1:6828/445214574,v1:127.0.0.1:6829/445214574] exists,up 2209119d-03bd-46eb-a070-528d29935b4e 2026-02-25T22:34:28.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:34:28.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:34:28.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:34:28.857 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:34:28.857 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:34:28.857 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:34:28.858 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:28.858 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:28.858 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:28.858 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:34:28.858 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:34:28.858 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:34:28.912 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:34:28.918 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:34:28.920 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:34:15.573+0000 7f03243dea40 0 load: jerasure load: lrc 2026-02-25T22:34:28.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:290: TEST_rados_get_subread_eio_shard_1: local poolname=pool-jerasure 2026-02-25T22:34:28.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:291: TEST_rados_get_subread_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T22:34:28.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:34:28.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:34:28.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T22:34:28.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:34:28.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T22:34:28.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:34:28.921 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T22:34:29.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:34:29.190 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:34:29.553 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:34:29.564 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:34:30.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:34:30.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:34:30.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:34:30.566 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:34:30.566 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:34:30.567 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:34:30.567 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:34:30.568 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:34:30.568 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:34:30.568 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:34:30.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:34:30.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:34:30.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:34:30.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:34:30.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:34:30.679 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:34:30.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:34:30.892 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:34:30.892 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:34:30.892 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:34:30.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:34:30.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:30.892 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:34:30.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803781 2026-02-25T22:34:30.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803781 2026-02-25T22:34:30.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781' 2026-02-25T22:34:30.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:30.973 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:34:31.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607556 2026-02-25T22:34:31.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607556 2026-02-25T22:34:31.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607556' 2026-02-25T22:34:31.050 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:31.050 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:34:31.130 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444037 2026-02-25T22:34:31.130 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444037 2026-02-25T22:34:31.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607556 2-73014444037' 2026-02-25T22:34:31.131 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:31.131 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:34:31.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182403 2026-02-25T22:34:31.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182403 2026-02-25T22:34:31.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607556 2-73014444037 3-107374182403' 2026-02-25T22:34:31.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:31.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803781 2026-02-25T22:34:31.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:31.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:34:31.211 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803781 2026-02-25T22:34:31.211 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:31.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803781 2026-02-25T22:34:31.213 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803781 2026-02-25T22:34:31.213 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803781' 2026-02-25T22:34:31.213 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:34:31.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803781 2026-02-25T22:34:31.426 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:31.426 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607556 2026-02-25T22:34:31.427 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:31.428 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:34:31.429 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607556 2026-02-25T22:34:31.429 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:31.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607556 2026-02-25T22:34:31.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607556' 2026-02-25T22:34:31.431 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 51539607556 2026-02-25T22:34:31.431 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:34:31.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607556 -lt 51539607556 2026-02-25T22:34:31.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:31.653 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444037 2026-02-25T22:34:31.653 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:31.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:34:31.655 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444037 2026-02-25T22:34:31.655 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:31.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444037 2026-02-25T22:34:31.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444037' 2026-02-25T22:34:31.657 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 73014444037 2026-02-25T22:34:31.657 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:34:31.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444037 -lt 73014444037 2026-02-25T22:34:31.873 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:31.874 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-107374182403 2026-02-25T22:34:31.874 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:31.875 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:34:31.876 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-107374182403 2026-02-25T22:34:31.876 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:31.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=107374182403 2026-02-25T22:34:31.878 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 107374182403' 2026-02-25T22:34:31.878 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 107374182403 2026-02-25T22:34:31.878 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:34:32.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 107374182401 -lt 107374182403 2026-02-25T22:34:32.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:34:33.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:34:33.088 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:34:33.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 107374182403 -lt 107374182403 2026-02-25T22:34:33.297 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:34:33.297 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:34:33.297 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:34:33.570 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:34:33.570 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:34:33.571 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:34:33.572 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:34:33.572 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:34:33.572 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:34:33.572 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:34:33.572 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:34:33.775 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:34:33.775 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:34:33.776 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:34:33.776 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:34:34.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:34:34.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:34:34.051 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:293: TEST_rados_get_subread_eio_shard_1: local shard_id=1 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:294: TEST_rados_get_subread_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-73931-1 2026-02-25T22:34:34.052 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-73931-1 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-73931-1 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:34:34.053 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-73931-1 td/test-erasure-eio/ORIGINAL 2026-02-25T22:34:34.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-73931-1 td/test-erasure-eio 1 2026-02-25T22:34:34.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:34:34.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:34:34.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:34:34.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:34:34.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:34:34.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:34:34.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-73931-1 2026-02-25T22:34:34.071 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:34:34.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:34:34.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:34:34.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T22:34:34.072 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:34:34.072 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-73931-1 2026-02-25T22:34:34.072 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:34:34.072 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-73931-1 2026-02-25T22:34:34.072 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-73931-1 2026-02-25T22:34:34.073 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:34:34.282 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:34:34.282 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:34:34.282 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:34:34.282 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:34:34.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T22:34:34.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:34:34.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=1 2026-02-25T22:34:34.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:34:34.283 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/1/type 2026-02-25T22:34:34.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:34:34.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-25T22:34:34.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:34:34.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-25T22:34:34.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:34:34.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:34:34.285 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:34:34.285 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-25T22:34:34.285 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:34:34.285 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:34:34.286 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:34.286 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:34.286 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:34.286 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.1.asok 2026-02-25T22:34:34.286 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:34:34.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:34:34.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:34:34.348 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:34:34.349 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.1 2026-02-25T22:34:34.349 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:34:34.349 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:34:34.350 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:34.350 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:34.350 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:34.350 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.1.asok 2026-02-25T22:34:34.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:34:34.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-73931-1 1 2026-02-25T22:34:34.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-73931-1 2026-02-25T22:34:34.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:34:34.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:34:34.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-73931-1 2026-02-25T22:34:34.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:34:34.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:34:34.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-73931-1 td/test-erasure-eio/COPY 2026-02-25T22:34:34.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:34:34.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:34:34.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2026-02-25T22:34:34.432 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2026-02-25T22:34:34.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=2 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-73931-1 td/test-erasure-eio 2 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-73931-1 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:34:34.435 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=2 2026-02-25T22:34:34.436 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:34:34.436 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-73931-1 2026-02-25T22:34:34.436 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:34:34.436 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-73931-1 2026-02-25T22:34:34.436 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-73931-1 2026-02-25T22:34:34.436 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:34:34.647 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:34:34.647 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:34:34.648 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:34:34.648 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:34:34.648 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T22:34:34.648 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:34:34.648 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=2 2026-02-25T22:34:34.648 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:34:34.649 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/2/type 2026-02-25T22:34:34.649 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:34:34.649 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2026-02-25T22:34:34.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:34:34.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=2 2026-02-25T22:34:34.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:34:34.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:34:34.650 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:34:34.651 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.2 2026-02-25T22:34:34.651 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2026-02-25T22:34:34.651 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2026-02-25T22:34:34.651 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:34.651 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:34.651 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:34.652 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.2.asok 2026-02-25T22:34:34.652 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:34:34.711 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:34:34.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:34:34.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:34:34.713 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.2 2026-02-25T22:34:34.713 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2026-02-25T22:34:34.713 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2026-02-25T22:34:34.714 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:34.714 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:34.714 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:34.714 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.2.asok 2026-02-25T22:34:34.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:34:34.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.2.asok injectdataerr pool-jerasure obj-eio-73931-1 2 2026-02-25T22:34:34.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 1 -eq 1 -a 2 -eq 2 ']' 2026-02-25T22:34:34.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:188: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-73931-1 2026-02-25T22:34:34.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:34:34.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:34:34.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-73931-1 2026-02-25T22:34:34.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:34:34.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:34:34.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-73931-1 td/test-erasure-eio/COPY 2026-02-25T22:34:34.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:34:34.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:34:34.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2026-02-25T22:34:34.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:295: TEST_rados_get_subread_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:34:34.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:34:34.796 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:34:35.053 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:34:35.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:34:35.363 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:34:35.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:34:35.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:34:35.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:34:35.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:34:35.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:34:35.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:34:35.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:34:35.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:34:35.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:34:35.484 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:34:35.485 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:34:35.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:34:35.486 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:34:35.487 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:34:35.488 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:34:35.488 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:34:35.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:34:35.489 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:34:35.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:34:35.490 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:34:35.491 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:34:35.492 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:34:35.492 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:34:35.505 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:34:35.506 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:35.506 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:35.506 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:34:35.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:34:35.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:34:35.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:34:35.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:34:35.508 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:34:35.509 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:34:35.509 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:34:35.509 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:34:35.509 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:34:35.509 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:34:35.509 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:34:35.509 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:34:35.509 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:34:35.509 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:34:35.511 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:34:35.511 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:34:35.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:34:35.512 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:34:35.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:34:35.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:34:35.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:34:35.515 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:34:35.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:34:35.516 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:34:35.516 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:34:35.517 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:34:35.519 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:34:35.519 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:34:35.520 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:34:35.520 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:35.520 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:35.520 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:34:35.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:34:35.521 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:34:35.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:34:35.523 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:34:35.523 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:35.523 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:35.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:34:35.525 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:34:35.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:34:35.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:34:35.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:34:35.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:34:35.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:34:35.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:34:35.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:34:35.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:34:35.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:34:35.527 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:34:35.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:34:35.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:35.547 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:35.548 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:35.548 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:35.548 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:35.548 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:35.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:34:35.578 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:34:35.579 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:34:35.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:34:35.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:34:35.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:34:35.580 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:34:35.580 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:34:35.580 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:34:35.580 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:34:35.581 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:35.581 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:35.581 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:35.582 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:34:35.582 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:34:35.582 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:34:35.636 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:34:35.636 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:34:35.636 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:34:35.637 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:34:35.638 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:34:35.638 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:34:35.638 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:34:35.638 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:34:35.638 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:35.638 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:35.638 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:35.638 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:34:35.639 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:34:35.639 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:34:35.693 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:34:35.693 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:34:35.694 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:34:35.694 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:34:35.694 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:34:35.694 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:34:35.694 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:34:35.805 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:34:35.805 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:35.805 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:35.806 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:35.806 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:35.806 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:35.806 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:35.806 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:34:35.807 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:34:35.825 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:34:35.825 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:34:35.940 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:34:35.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:34:36.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:34:36.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:34:36.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:34:36.951 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:36.951 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:36.951 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:36.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:34:36.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:34:36.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:34:37.000 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:34:37.005 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:34:37.007 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:34:35.565+0000 7f935db2dec0 0 load: jerasure load: lrc 2026-02-25T22:34:37.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_missing td/test-erasure-eio 2026-02-25T22:34:37.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_subread_missing: local dir=td/test-erasure-eio 2026-02-25T22:34:37.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_subread_missing: setup_osds 4 2026-02-25T22:34:37.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T22:34:37.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:34:37.009 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T22:34:37.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T22:34:37.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:34:37.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:34:37.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:34:37.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:34:37.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:34:37.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:34:37.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:34:37.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:34:37.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:34:37.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:34:37.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:37.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:34:37.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:34:37.017 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:34:37.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d4e8d043-5410-412f-8453-4c60a6048945 2026-02-25T22:34:37.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d4e8d043-5410-412f-8453-4c60a6048945' 2026-02-25T22:34:37.018 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 d4e8d043-5410-412f-8453-4c60a6048945 2026-02-25T22:34:37.018 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:34:37.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD9eJ9pTUd5ARAAtLr23jokZoxjFgFffT/AJw== 2026-02-25T22:34:37.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD9eJ9pTUd5ARAAtLr23jokZoxjFgFffT/AJw=="}' 2026-02-25T22:34:37.026 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d4e8d043-5410-412f-8453-4c60a6048945 -i td/test-erasure-eio/0/new.json 2026-02-25T22:34:37.140 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:34:37.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:34:37.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9eJ9pTUd5ARAAtLr23jokZoxjFgFffT/AJw== --osd-uuid d4e8d043-5410-412f-8453-4c60a6048945 2026-02-25T22:34:37.166 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:37.161+0000 7f1c6dbf6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:37.167 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:37.161+0000 7f1c6dbf6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:37.168 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:37.165+0000 7f1c6dbf6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:37.169 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:37.165+0000 7f1c6dbf6a40 -1 bdev(0x557c66ff0400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:34:37.169 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:37.165+0000 7f1c6dbf6a40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:34:37.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:34:37.905 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:34:37.906 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:34:37.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:34:37.906 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:34:38.039 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:34:38.040 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:34:38.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:34:38.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:34:38.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:34:38.040 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:34:38.055 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:38.049+0000 7f1e87890a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:38.056 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:38.053+0000 7f1e87890a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:38.057 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:38.053+0000 7f1e87890a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:38.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:34:38.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:34:38.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:34:38.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:34:38.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:34:38.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:38.149 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:34:38.150 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:34:38.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:38.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:34:38.360 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:38.433 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:38.429+0000 7f1e87890a40 -1 Falling back to public interface 2026-02-25T22:34:39.084 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:39.081+0000 7f1e87890a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:34:39.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:39.362 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:39.362 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:39.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:34:39.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:39.363 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:34:39.577 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:40.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:40.579 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:40.579 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:34:40.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:34:40.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:40.580 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:34:40.787 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2469244918,v1:127.0.0.1:6803/2469244918] [v2:127.0.0.1:6804/2469244918,v1:127.0.0.1:6805/2469244918] exists,up d4e8d043-5410-412f-8453-4c60a6048945 2026-02-25T22:34:40.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:34:40.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:34:40.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:34:40.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:34:40.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:34:40.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:34:40.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:34:40.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:34:40.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:34:40.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:40.792 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:34:40.793 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:34:40.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:34:40.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:34:40.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:34:40.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:34:40.794 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:34:40.794 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:34:40.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f6d61a3d-3bd7-45e9-a1d4-6f77cd85e007 2026-02-25T22:34:40.795 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 f6d61a3d-3bd7-45e9-a1d4-6f77cd85e007 2026-02-25T22:34:40.795 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f6d61a3d-3bd7-45e9-a1d4-6f77cd85e007' 2026-02-25T22:34:40.795 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:34:40.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAAeZ9pPTzuLxAA7QsbvU8Te2DLC0YHOhjLfg== 2026-02-25T22:34:40.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAAeZ9pPTzuLxAA7QsbvU8Te2DLC0YHOhjLfg=="}' 2026-02-25T22:34:40.806 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f6d61a3d-3bd7-45e9-a1d4-6f77cd85e007 -i td/test-erasure-eio/1/new.json 2026-02-25T22:34:41.020 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:41.029 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:34:41.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAAeZ9pPTzuLxAA7QsbvU8Te2DLC0YHOhjLfg== --osd-uuid f6d61a3d-3bd7-45e9-a1d4-6f77cd85e007 2026-02-25T22:34:41.047 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:41.041+0000 7ffacb37ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:41.049 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:41.045+0000 7ffacb37ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:41.050 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:41.045+0000 7ffacb37ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:41.050 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:41.045+0000 7ffacb37ca40 -1 bdev(0x562e92690400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:34:41.051 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:41.045+0000 7ffacb37ca40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:34:41.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:34:41.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:34:41.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:34:41.728 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:34:41.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:34:42.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:34:42.009 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:34:42.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:34:42.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:34:42.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:34:42.010 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:34:42.032 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:42.029+0000 7f8763f7ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:42.034 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:42.029+0000 7f8763f7ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:42.035 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:42.029+0000 7f8763f7ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:42.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:34:42.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:34:42.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:34:42.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:34:42.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:34:42.224 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:42.224 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:34:42.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:34:42.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:42.225 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:34:42.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:42.509 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:42.505+0000 7f8763f7ca40 -1 Falling back to public interface 2026-02-25T22:34:43.105 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:43.101+0000 7f8763f7ca40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:34:43.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:43.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:43.433 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:43.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:34:43.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:43.434 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:34:43.649 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:44.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:44.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:44.651 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:34:44.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:34:44.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:44.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:34:44.859 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3719033079,v1:127.0.0.1:6811/3719033079] [v2:127.0.0.1:6812/3719033079,v1:127.0.0.1:6813/3719033079] exists,up f6d61a3d-3bd7-45e9-a1d4-6f77cd85e007 2026-02-25T22:34:44.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:34:44.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:34:44.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:34:44.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:34:44.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:34:44.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:34:44.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:34:44.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:34:44.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:34:44.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:34:44.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:34:44.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:34:44.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:34:44.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:34:44.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:44.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:34:44.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:34:44.866 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:34:44.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=109a5a4a-152c-4c24-9fcd-7a3da97cc750 2026-02-25T22:34:44.866 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 109a5a4a-152c-4c24-9fcd-7a3da97cc750 2026-02-25T22:34:44.867 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 109a5a4a-152c-4c24-9fcd-7a3da97cc750' 2026-02-25T22:34:44.867 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:34:44.876 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAEeZ9pDdQbNBAAVTa/cz1stcKdnchM1cDnhA== 2026-02-25T22:34:44.876 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAEeZ9pDdQbNBAAVTa/cz1stcKdnchM1cDnhA=="}' 2026-02-25T22:34:44.876 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 109a5a4a-152c-4c24-9fcd-7a3da97cc750 -i td/test-erasure-eio/2/new.json 2026-02-25T22:34:45.086 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:34:45.097 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:34:45.098 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEeZ9pDdQbNBAAVTa/cz1stcKdnchM1cDnhA== --osd-uuid 109a5a4a-152c-4c24-9fcd-7a3da97cc750 2026-02-25T22:34:45.121 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:45.117+0000 7fb41b113a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:45.123 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:45.121+0000 7fb41b113a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:45.124 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:45.121+0000 7fb41b113a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:45.125 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:45.121+0000 7fb41b113a40 -1 bdev(0x55f94e3e0400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:34:45.125 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:45.121+0000 7fb41b113a40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:34:45.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:34:45.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:34:45.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:34:45.852 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:34:45.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:34:46.137 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:34:46.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:34:46.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:34:46.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:34:46.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:34:46.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:34:46.160 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:46.157+0000 7f98dad98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:46.161 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:46.157+0000 7f98dad98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:46.163 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:46.157+0000 7f98dad98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:46.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:34:46.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:34:46.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:34:46.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:34:46.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:34:46.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:46.351 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:34:46.351 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:34:46.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:46.352 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:34:46.560 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:46.561 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:46.557+0000 7f98dad98a40 -1 Falling back to public interface 2026-02-25T22:34:47.209 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:47.205+0000 7f98dad98a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:34:47.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:47.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:47.561 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:34:47.562 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:47.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:47.562 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:34:47.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:48.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:48.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:48.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:34:48.772 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:34:48.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:48.773 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:34:48.978 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 18 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2210126693,v1:127.0.0.1:6819/2210126693] [v2:127.0.0.1:6820/2210126693,v1:127.0.0.1:6821/2210126693] exists,up 109a5a4a-152c-4c24-9fcd-7a3da97cc750 2026-02-25T22:34:48.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:34:48.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:34:48.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:34:48.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:34:48.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:34:48.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:34:48.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:34:48.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:34:48.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:34:48.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:34:48.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:34:48.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:34:48.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:34:48.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:34:48.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:34:48.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:34:48.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:34:48.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:34:48.983 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:34:48.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:34:48.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:34:48.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:34:48.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:34:48.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:34:48.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:34:48.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:34:48.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:34:48.984 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:34:48.985 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:34:48.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bc43de67-1cae-4a2f-b668-a716533f6d12 2026-02-25T22:34:48.985 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 bc43de67-1cae-4a2f-b668-a716533f6d12' 2026-02-25T22:34:48.985 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 bc43de67-1cae-4a2f-b668-a716533f6d12 2026-02-25T22:34:48.986 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:34:48.995 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAIeZ9pDQY5OxAA+UOKypaWHHl8JDHRbdeidw== 2026-02-25T22:34:48.995 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAIeZ9pDQY5OxAA+UOKypaWHHl8JDHRbdeidw=="}' 2026-02-25T22:34:48.995 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bc43de67-1cae-4a2f-b668-a716533f6d12 -i td/test-erasure-eio/3/new.json 2026-02-25T22:34:49.211 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:34:49.220 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:34:49.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIeZ9pDQY5OxAA+UOKypaWHHl8JDHRbdeidw== --osd-uuid bc43de67-1cae-4a2f-b668-a716533f6d12 2026-02-25T22:34:49.237 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:49.233+0000 7f4e20c88a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:49.238 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:49.233+0000 7f4e20c88a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:49.239 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:49.233+0000 7f4e20c88a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:49.240 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:49.237+0000 7f4e20c88a40 -1 bdev(0x556225a14400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:34:49.240 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:49.237+0000 7f4e20c88a40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:34:49.902 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:34:49.902 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:34:49.903 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:34:49.903 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:34:49.904 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:34:50.194 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:34:50.195 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:34:50.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:34:50.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:34:50.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:34:50.196 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:34:50.210 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:50.205+0000 7f9375211a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:50.211 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:50.209+0000 7f9375211a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:50.212 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:50.209+0000 7f9375211a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:50.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:34:50.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:34:50.401 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:34:50.401 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:34:50.401 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:34:50.401 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:50.401 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:34:50.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:34:50.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:50.402 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:34:50.610 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:50.689 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:50.685+0000 7f9375211a40 -1 Falling back to public interface 2026-02-25T22:34:51.304 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:51.301+0000 7f9375211a40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:34:51.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:51.612 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:51.612 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:34:51.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:34:51.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:51.613 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:34:51.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:34:52.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:34:52.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:52.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:34:52.847 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:34:52.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:52.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:34:53.061 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 26 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1075378911,v1:127.0.0.1:6827/1075378911] [v2:127.0.0.1:6828/1075378911,v1:127.0.0.1:6829/1075378911] exists,up bc43de67-1cae-4a2f-b668-a716533f6d12 2026-02-25T22:34:53.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:34:53.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:34:53.062 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:34:53.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:34:53.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:34:53.063 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:34:53.064 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:34:53.064 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:53.064 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:53.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:34:53.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:34:53.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:34:53.120 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:34:53.126 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:34:53.127 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:34:38.433+0000 7f1e87890a40 0 load: jerasure load: lrc 2026-02-25T22:34:53.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_subread_missing: local poolname=pool-jerasure 2026-02-25T22:34:53.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_subread_missing: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T22:34:53.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:34:53.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:34:53.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T22:34:53.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:34:53.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T22:34:53.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:34:53.129 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T22:34:53.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:34:53.397 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:34:53.761 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:34:53.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:34:54.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:34:54.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:34:54.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:34:54.772 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:34:54.773 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:34:54.773 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:34:54.773 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:34:54.774 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:34:54.774 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:34:54.774 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:34:54.884 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:34:54.884 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:34:54.884 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:34:54.884 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:34:54.884 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:34:54.884 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:34:55.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:34:55.100 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:34:55.100 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:34:55.100 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:34:55.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:34:55.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:55.100 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:34:55.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803781 2026-02-25T22:34:55.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803781 2026-02-25T22:34:55.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781' 2026-02-25T22:34:55.178 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:55.178 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:34:55.255 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574853 2026-02-25T22:34:55.255 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574853 2026-02-25T22:34:55.255 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-55834574853' 2026-02-25T22:34:55.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:55.256 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:34:55.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411332 2026-02-25T22:34:55.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411332 2026-02-25T22:34:55.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-55834574853 2-77309411332' 2026-02-25T22:34:55.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:34:55.337 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:34:55.417 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149699 2026-02-25T22:34:55.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149699 2026-02-25T22:34:55.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-55834574853 2-77309411332 3-111669149699' 2026-02-25T22:34:55.418 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:55.418 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803781 2026-02-25T22:34:55.419 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:55.419 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:34:55.420 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803781 2026-02-25T22:34:55.420 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:55.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803781 2026-02-25T22:34:55.422 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803781 2026-02-25T22:34:55.422 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803781' 2026-02-25T22:34:55.423 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:34:55.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803779 -lt 25769803781 2026-02-25T22:34:55.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:34:56.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:34:56.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:34:56.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803781 2026-02-25T22:34:56.854 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:56.854 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574853 2026-02-25T22:34:56.854 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:56.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:34:56.857 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574853 2026-02-25T22:34:56.857 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:56.858 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574853 2026-02-25T22:34:56.858 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 55834574853 2026-02-25T22:34:56.859 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574853' 2026-02-25T22:34:56.859 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:34:57.079 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574853 -lt 55834574853 2026-02-25T22:34:57.079 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:57.080 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411332 2026-02-25T22:34:57.080 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:57.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:34:57.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411332 2026-02-25T22:34:57.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:57.084 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411332 2026-02-25T22:34:57.084 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 77309411332 2026-02-25T22:34:57.084 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411332' 2026-02-25T22:34:57.085 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:34:57.292 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411332 -lt 77309411332 2026-02-25T22:34:57.292 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:34:57.293 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149699 2026-02-25T22:34:57.293 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:34:57.294 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:34:57.294 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149699 2026-02-25T22:34:57.295 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:34:57.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149699 2026-02-25T22:34:57.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149699' 2026-02-25T22:34:57.297 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149699 2026-02-25T22:34:57.297 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:34:57.502 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149699 -lt 111669149699 2026-02-25T22:34:57.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:34:57.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:34:57.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:34:57.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:34:57.779 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:34:57.780 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:34:57.780 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:34:57.780 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:34:57.780 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:34:57.780 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:34:57.781 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:34:57.983 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:34:57.984 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:34:57.985 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:34:57.985 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:34:58.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:34:58.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:34:58.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:34:58.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_subread_missing: local shard_id=1 2026-02-25T22:34:58.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_subread_missing: rados_put_get_data remove td/test-erasure-eio 1 2026-02-25T22:34:58.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=remove 2026-02-25T22:34:58.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-25T22:34:58.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-25T22:34:58.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-25T22:34:58.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2026-02-25T22:34:58.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-25T22:34:58.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2026-02-25T22:34:58.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-25T22:34:58.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-remove-73931-1 2026-02-25T22:34:58.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-remove-73931-1 2026-02-25T22:34:58.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:34:58.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:34:58.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-remove-73931-1 2026-02-25T22:34:58.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:58.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:34:58.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:58.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:34:58.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:58.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:34:58.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:34:58.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:34:58.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-remove-73931-1 td/test-erasure-eio/ORIGINAL 2026-02-25T22:34:58.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-73931-1 td/test-erasure-eio 1 2026-02-25T22:34:58.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2026-02-25T22:34:58.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2026-02-25T22:34:58.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2026-02-25T22:34:58.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2026-02-25T22:34:58.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2026-02-25T22:34:58.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2026-02-25T22:34:58.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-73931-1 2026-02-25T22:34:58.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2026-02-25T22:34:58.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2026-02-25T22:34:58.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2026-02-25T22:34:58.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=1 2026-02-25T22:34:58.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2026-02-25T22:34:58.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-73931-1 2026-02-25T22:34:58.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:34:58.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-remove-73931-1 2026-02-25T22:34:58.285 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-remove-73931-1 2026-02-25T22:34:58.285 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:34:58.496 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:34:58.496 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:34:58.496 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:34:58.496 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:34:58.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2026-02-25T22:34:58.498 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2026-02-25T22:34:58.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=1 2026-02-25T22:34:58.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 1 obj-remove-73931-1 remove 2026-02-25T22:34:58.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:34:58.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:34:58.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T22:34:58.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:34:58.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-remove-73931-1 remove 2026-02-25T22:34:58.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:34:58.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:34:58.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T22:34:58.499 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:34:58.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T22:34:58.500 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:34:58.500 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:34:58.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:34:58.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:34:58.500 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:34:58.806 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:34:58.806 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-remove-73931-1 remove 2026-02-25T22:34:58.806 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:34:58.807 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:34:58.807 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T22:34:58.807 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:34:58.807 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T22:34:58.807 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-remove-73931-1 remove 2026-02-25T22:34:59.023 INFO:tasks.workunit.client.0.trial125.stdout:remove 1#2:7fb87f68:::obj-remove-73931-1:head# 2026-02-25T22:34:59.114 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T22:34:59.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:34:59.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:34:59.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T22:34:59.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:34:59.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:34:59.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:34:59.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:34:59.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:34:59.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:34:59.115 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:59.116 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:34:59.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:34:59.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:34:59.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:34:59.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:34:59.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:34:59.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:34:59.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:34:59.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:34:59.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:34:59.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:34:59.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T22:34:59.119 INFO:tasks.workunit.client.0.trial125.stderr:start osd.1 2026-02-25T22:34:59.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:34:59.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T22:34:59.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T22:34:59.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:34:59.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:34:59.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:34:59.138 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:59.132+0000 7fed41972a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:59.140 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:59.136+0000 7fed41972a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:59.141 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:59.136+0000 7fed41972a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:34:59.330 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T22:34:59.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:34:59.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:34:59.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:34:59.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:34:59.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:34:59.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:34:59.331 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:34:59.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:34:59.331 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:34:59.481 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:34:59.476+0000 7fed41972a40 -1 Falling back to public interface 2026-02-25T22:34:59.545 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:00.097 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:00.092+0000 7fed41972a40 -1 osd.1 34 log_to_monitors true 2026-02-25T22:35:00.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:00.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:00.547 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:35:00.547 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:35:00.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:00.548 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:35:00.761 INFO:tasks.workunit.client.0.trial125.stderr:osd.1 up in weight 1 up_from 37 up_thru 37 down_at 34 last_clean_interval [13,34) [v2:127.0.0.1:6810/1970518209,v1:127.0.0.1:6811/1970518209] [v2:127.0.0.1:6812/1970518209,v1:127.0.0.1:6813/1970518209] exists,up f6d61a3d-3bd7-45e9-a1d4-6f77cd85e007 2026-02-25T22:35:00.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:35:00.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:35:00.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:35:00.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:35:00.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:35:00.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:35:00.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:35:00.763 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:35:00.763 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:35:00.763 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:35:00.764 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:35:00.764 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:35:00.764 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:35:00.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:35:00.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:35:00.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:35:00.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:35:00.874 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:35:00.874 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:35:01.084 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:35:01.085 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:35:01.085 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:35:01.085 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:35:01.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:35:01.085 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:01.085 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:35:01.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803785 2026-02-25T22:35:01.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803785 2026-02-25T22:35:01.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785' 2026-02-25T22:35:01.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:01.167 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:35:01.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789955 2026-02-25T22:35:01.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789955 2026-02-25T22:35:01.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-158913789955' 2026-02-25T22:35:01.247 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:01.247 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:35:01.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411335 2026-02-25T22:35:01.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411335 2026-02-25T22:35:01.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-158913789955 2-77309411335' 2026-02-25T22:35:01.327 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:01.328 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:35:01.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149702 2026-02-25T22:35:01.406 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149702 2026-02-25T22:35:01.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-158913789955 2-77309411335 3-111669149702' 2026-02-25T22:35:01.407 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:01.407 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803785 2026-02-25T22:35:01.407 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:01.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:35:01.409 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803785 2026-02-25T22:35:01.409 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:01.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803785 2026-02-25T22:35:01.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2026-02-25T22:35:01.411 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803785 2026-02-25T22:35:01.411 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:35:01.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803785 2026-02-25T22:35:01.634 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:35:02.635 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:35:02.636 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:35:02.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803785 2026-02-25T22:35:02.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:02.843 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-158913789955 2026-02-25T22:35:02.843 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:02.844 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:35:02.845 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-158913789955 2026-02-25T22:35:02.846 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:02.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=158913789955 2026-02-25T22:35:02.847 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 158913789955' 2026-02-25T22:35:02.847 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 158913789955 2026-02-25T22:35:02.848 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:35:03.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 158913789955 -lt 158913789955 2026-02-25T22:35:03.064 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:03.065 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411335 2026-02-25T22:35:03.065 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:03.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:35:03.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411335 2026-02-25T22:35:03.067 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:03.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411335 2026-02-25T22:35:03.068 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411335' 2026-02-25T22:35:03.068 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 77309411335 2026-02-25T22:35:03.069 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:35:03.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411335 -lt 77309411335 2026-02-25T22:35:03.282 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:03.283 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149702 2026-02-25T22:35:03.283 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:03.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:35:03.285 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149702 2026-02-25T22:35:03.285 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:03.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149702 2026-02-25T22:35:03.287 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149702' 2026-02-25T22:35:03.287 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 111669149702 2026-02-25T22:35:03.287 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:35:03.503 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149702 -lt 111669149702 2026-02-25T22:35:03.503 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:35:03.504 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:35:03.504 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:35:03.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:35:03.778 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:35:03.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:35:03.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:35:03.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:35:03.779 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:35:03.780 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:35:03.780 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:35:03.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:35:03.981 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:35:03.981 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:35:03.982 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:35:04.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:35:04.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:35:04.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:35:04.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-73931-1 2026-02-25T22:35:04.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:35:04.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:35:04.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-73931-1 2026-02-25T22:35:04.260 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:35:04.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:35:04.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-73931-1 td/test-erasure-eio/COPY 2026-02-25T22:35:04.278 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:35:04.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:35:04.280 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2026-02-25T22:35:04.280 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2026-02-25T22:35:04.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=2 2026-02-25T22:35:04.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-73931-1 td/test-erasure-eio 2 2026-02-25T22:35:04.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2026-02-25T22:35:04.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2026-02-25T22:35:04.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2026-02-25T22:35:04.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2026-02-25T22:35:04.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2026-02-25T22:35:04.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2026-02-25T22:35:04.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-73931-1 2026-02-25T22:35:04.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2026-02-25T22:35:04.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2026-02-25T22:35:04.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2026-02-25T22:35:04.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=2 2026-02-25T22:35:04.284 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2026-02-25T22:35:04.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-73931-1 2026-02-25T22:35:04.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:35:04.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-remove-73931-1 2026-02-25T22:35:04.285 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-remove-73931-1 2026-02-25T22:35:04.285 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:35:04.494 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:35:04.494 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:35:04.494 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:35:04.494 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:35:04.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2026-02-25T22:35:04.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2026-02-25T22:35:04.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=2 2026-02-25T22:35:04.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 2 obj-remove-73931-1 remove 2026-02-25T22:35:04.494 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-remove-73931-1 remove 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:35:04.495 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:35:04.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:35:04.496 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:35:04.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:35:04.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-remove-73931-1 remove 2026-02-25T22:35:04.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T22:35:04.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:35:04.803 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-25T22:35:04.804 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:35:04.804 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-25T22:35:04.804 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-remove-73931-1 remove 2026-02-25T22:35:05.019 INFO:tasks.workunit.client.0.trial125.stdout:remove 2#2:7fb87f68:::obj-remove-73931-1:head# 2026-02-25T22:35:05.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-25T22:35:05.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:35:05.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:35:05.117 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T22:35:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:35:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:35:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:35:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:35:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:35:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:35:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:35:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:35:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:35:05.118 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:35:05.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:35:05.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:35:05.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:05.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:05.119 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:05.119 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:05.119 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:05.119 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:05.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:05.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:35:05.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:35:05.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:35:05.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:35:05.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:35:05.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:35:05.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:35:05.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:35:05.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:35:05.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:35:05.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T22:35:05.122 INFO:tasks.workunit.client.0.trial125.stderr:start osd.2 2026-02-25T22:35:05.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:35:05.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T22:35:05.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T22:35:05.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:35:05.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:35:05.123 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:35:05.142 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:05.136+0000 7fb72b7eea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:05.143 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:05.136+0000 7fb72b7eea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:05.144 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:05.140+0000 7fb72b7eea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:05.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T22:35:05.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:35:05.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:35:05.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:35:05.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:35:05.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:05.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:35:05.336 INFO:tasks.workunit.client.0.trial125.stderr:0 2026-02-25T22:35:05.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:05.336 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:35:05.550 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:05.637 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:05.632+0000 7fb72b7eea40 -1 Falling back to public interface 2026-02-25T22:35:06.213 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:06.208+0000 7fb72b7eea40 -1 osd.2 40 log_to_monitors true 2026-02-25T22:35:06.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:06.552 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:06.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:35:06.553 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:35:06.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:06.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:35:06.766 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 up in weight 1 up_from 43 up_thru 34 down_at 40 last_clean_interval [18,40) [v2:127.0.0.1:6818/2425579924,v1:127.0.0.1:6819/2425579924] [v2:127.0.0.1:6820/2425579924,v1:127.0.0.1:6821/2425579924] exists,up 109a5a4a-152c-4c24-9fcd-7a3da97cc750 2026-02-25T22:35:06.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:35:06.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:35:06.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:35:06.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:35:06.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:35:06.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:35:06.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:35:06.767 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:35:06.768 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:35:06.768 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:35:06.768 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:35:06.768 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:35:06.769 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:35:06.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:35:06.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:35:06.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:35:06.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:35:06.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:35:06.880 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:35:07.099 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:35:07.100 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:35:07.100 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:35:07.100 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:35:07.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:35:07.100 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:07.100 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:35:07.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803788 2026-02-25T22:35:07.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803788 2026-02-25T22:35:07.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788' 2026-02-25T22:35:07.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:07.177 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:35:07.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789958 2026-02-25T22:35:07.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789958 2026-02-25T22:35:07.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-158913789958' 2026-02-25T22:35:07.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:07.257 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:35:07.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593731 2026-02-25T22:35:07.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593731 2026-02-25T22:35:07.334 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-158913789958 2-184683593731' 2026-02-25T22:35:07.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:07.335 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:35:07.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149705 2026-02-25T22:35:07.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149705 2026-02-25T22:35:07.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803788 1-158913789958 2-184683593731 3-111669149705' 2026-02-25T22:35:07.412 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:07.413 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803788 2026-02-25T22:35:07.413 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:07.414 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:35:07.415 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803788 2026-02-25T22:35:07.415 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:07.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803788 2026-02-25T22:35:07.416 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803788' 2026-02-25T22:35:07.416 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.0 seq 25769803788 2026-02-25T22:35:07.417 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:35:07.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803786 -lt 25769803788 2026-02-25T22:35:07.630 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:35:08.631 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:35:08.632 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:35:08.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803788 -lt 25769803788 2026-02-25T22:35:08.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:08.846 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-158913789958 2026-02-25T22:35:08.846 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:08.848 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:35:08.849 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-158913789958 2026-02-25T22:35:08.849 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:08.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=158913789958 2026-02-25T22:35:08.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 158913789958' 2026-02-25T22:35:08.850 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.1 seq 158913789958 2026-02-25T22:35:08.851 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:35:09.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 158913789958 -lt 158913789958 2026-02-25T22:35:09.061 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:09.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-184683593731 2026-02-25T22:35:09.062 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:09.063 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:35:09.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-184683593731 2026-02-25T22:35:09.064 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:09.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593731 2026-02-25T22:35:09.066 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 184683593731' 2026-02-25T22:35:09.066 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.2 seq 184683593731 2026-02-25T22:35:09.066 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:35:09.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593731 -lt 184683593731 2026-02-25T22:35:09.275 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:09.275 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149705 2026-02-25T22:35:09.276 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:09.277 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:35:09.277 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149705 2026-02-25T22:35:09.278 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:09.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149705 2026-02-25T22:35:09.279 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149705' 2026-02-25T22:35:09.279 INFO:tasks.workunit.client.0.trial125.stderr:waiting osd.3 seq 111669149705 2026-02-25T22:35:09.280 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:35:09.490 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149706 -lt 111669149705 2026-02-25T22:35:09.491 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:35:09.491 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:35:09.491 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:35:09.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:35:09.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:35:09.770 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:35:09.770 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:35:09.771 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:35:09.771 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:35:09.771 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:35:09.771 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:35:09.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:35:09.970 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:35:09.970 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:35:09.971 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:35:10.243 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:35:10.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:35:10.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:35:10.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 1 -eq 1 -a 2 -eq 2 ']' 2026-02-25T22:35:10.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:188: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-73931-1 2026-02-25T22:35:10.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:35:10.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:35:10.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-73931-1 2026-02-25T22:35:10.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:35:10.244 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:35:10.245 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-73931-1 td/test-erasure-eio/COPY 2026-02-25T22:35:10.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:35:10.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:35:10.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2026-02-25T22:35:10.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_subread_missing: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:35:10.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:35:10.265 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:35:10.525 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:35:10.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:35:10.811 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:35:10.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:35:10.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:35:10.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:35:10.821 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:35:10.822 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:35:10.822 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:35:10.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:35:10.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:35:10.822 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:35:10.932 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:35:10.933 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:35:10.934 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:35:10.935 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:35:10.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:35:10.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:35:10.937 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:35:10.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:35:10.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:35:10.938 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:35:10.939 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:35:10.940 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:35:10.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:35:10.941 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:35:10.955 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:35:10.955 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:10.955 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:10.955 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:35:10.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:35:10.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:35:10.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:35:10.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:35:10.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:35:10.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:35:10.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:35:10.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:35:10.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:35:10.958 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:35:10.958 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:35:10.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:35:10.958 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:35:10.959 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:35:10.960 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:35:10.960 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:35:10.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:35:10.961 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:35:10.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:35:10.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:35:10.963 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:35:10.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:35:10.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:35:10.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:35:10.965 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:35:10.966 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:35:10.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:35:10.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:35:10.969 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:35:10.969 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:10.969 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:10.969 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:35:10.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:35:10.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:35:10.971 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:35:10.972 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:35:10.972 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:10.972 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:10.972 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:35:10.974 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:35:10.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:35:10.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:35:10.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:35:10.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:35:10.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:35:10.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:35:10.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:35:10.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:35:10.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:35:10.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:35:10.996 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:35:10.996 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:10.997 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:10.997 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:10.997 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:10.997 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:10.997 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:10.998 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:35:11.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:35:11.022 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:35:11.023 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:35:11.023 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:35:11.023 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:35:11.023 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:35:11.024 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:35:11.024 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:35:11.024 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:35:11.024 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:35:11.025 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:11.025 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:11.025 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:35:11.025 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:35:11.025 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:35:11.080 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:35:11.080 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:35:11.080 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:35:11.080 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:35:11.080 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:35:11.080 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:35:11.080 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:35:11.081 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:35:11.081 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:35:11.081 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:11.081 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:11.081 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:35:11.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:35:11.082 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:35:11.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:35:11.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:35:11.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:35:11.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:35:11.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:35:11.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:35:11.138 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:35:11.248 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:35:11.248 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:11.248 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:11.249 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:11.249 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:11.249 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:11.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:11.250 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:35:11.250 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:35:11.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:35:11.269 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:35:11.385 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:35:11.393 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:35:12.395 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:35:12.395 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:35:12.395 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:35:12.395 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:35:12.395 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:12.396 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:12.396 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:35:12.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:35:12.396 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:35:12.444 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:35:12.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:35:12.451 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:35:11.008+0000 7fa997854ec0 0 load: jerasure load: lrc 2026-02-25T22:35:12.452 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_0 td/test-erasure-eio 2026-02-25T22:35:12.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:350: TEST_rados_get_with_subreadall_eio_shard_0: local dir=td/test-erasure-eio 2026-02-25T22:35:12.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:351: TEST_rados_get_with_subreadall_eio_shard_0: local shard_id=0 2026-02-25T22:35:12.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:353: TEST_rados_get_with_subreadall_eio_shard_0: setup_osds 4 2026-02-25T22:35:12.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T22:35:12.453 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:35:12.453 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T22:35:12.454 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T22:35:12.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:35:12.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:35:12.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:35:12.457 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:35:12.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:35:12.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:35:12.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:35:12.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:35:12.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:35:12.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:35:12.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:35:12.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:35:12.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:35:12.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:35:12.458 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:35:12.459 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:35:12.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:35:12.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:35:12.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:35:12.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:35:12.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:35:12.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:35:12.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:35:12.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:35:12.460 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:35:12.461 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:35:12.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e1801d21-bfb7-488f-9d9e-5d54b09abfc7 2026-02-25T22:35:12.461 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e1801d21-bfb7-488f-9d9e-5d54b09abfc7' 2026-02-25T22:35:12.461 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 e1801d21-bfb7-488f-9d9e-5d54b09abfc7 2026-02-25T22:35:12.462 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:35:12.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAgeZ9pnrwdHBAA4WyZT8wM6umaTX+9GxDA0A== 2026-02-25T22:35:12.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAgeZ9pnrwdHBAA4WyZT8wM6umaTX+9GxDA0A=="}' 2026-02-25T22:35:12.473 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e1801d21-bfb7-488f-9d9e-5d54b09abfc7 -i td/test-erasure-eio/0/new.json 2026-02-25T22:35:12.583 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:35:12.590 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:35:12.592 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgeZ9pnrwdHBAA4WyZT8wM6umaTX+9GxDA0A== --osd-uuid e1801d21-bfb7-488f-9d9e-5d54b09abfc7 2026-02-25T22:35:12.608 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:12.604+0000 7fc032e1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:12.609 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:12.604+0000 7fc032e1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:12.610 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:12.604+0000 7fc032e1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:12.611 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:12.604+0000 7fc032e1fa40 -1 bdev(0x564bd19be400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:35:12.611 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:12.604+0000 7fc032e1fa40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:35:13.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:35:13.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:35:13.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:35:13.264 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:35:13.264 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:35:13.378 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:35:13.379 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:35:13.379 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:35:13.380 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:35:13.391 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:35:13.403 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:35:13.445 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:13.436+0000 7f3f0ab33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:13.446 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:13.440+0000 7f3f0ab33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:13.447 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:13.440+0000 7f3f0ab33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:13.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:35:13.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:35:13.534 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:35:13.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:35:13.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:35:13.535 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:13.535 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:35:13.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:35:13.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:13.536 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:35:13.641 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:13.837 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:13.832+0000 7f3f0ab33a40 -1 Falling back to public interface 2026-02-25T22:35:14.381 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:14.376+0000 7f3f0ab33a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:35:14.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:14.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:14.643 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:35:14.644 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:35:14.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:14.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:35:14.854 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:15.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:15.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:15.856 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:35:15.856 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:35:15.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:15.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:35:16.069 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3365307409,v1:127.0.0.1:6803/3365307409] [v2:127.0.0.1:6804/3365307409,v1:127.0.0.1:6805/3365307409] exists,up e1801d21-bfb7-488f-9d9e-5d54b09abfc7 2026-02-25T22:35:16.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:35:16.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:35:16.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:35:16.073 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:35:16.074 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:16.075 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:35:16.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:35:16.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:35:16.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:35:16.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:35:16.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:35:16.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:35:16.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:35:16.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:35:16.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:35:16.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:35:16.076 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:35:16.077 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:35:16.077 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3cda0b6b-55fe-47b0-beda-a7288d10b693 2026-02-25T22:35:16.077 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 3cda0b6b-55fe-47b0-beda-a7288d10b693 2026-02-25T22:35:16.078 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3cda0b6b-55fe-47b0-beda-a7288d10b693' 2026-02-25T22:35:16.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:35:16.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAkeZ9prUQcBRAA5piqbI9P/A6Kkra6IzekKQ== 2026-02-25T22:35:16.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAkeZ9prUQcBRAA5piqbI9P/A6Kkra6IzekKQ=="}' 2026-02-25T22:35:16.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3cda0b6b-55fe-47b0-beda-a7288d10b693 -i td/test-erasure-eio/1/new.json 2026-02-25T22:35:16.302 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:35:16.312 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:35:16.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkeZ9prUQcBRAA5piqbI9P/A6Kkra6IzekKQ== --osd-uuid 3cda0b6b-55fe-47b0-beda-a7288d10b693 2026-02-25T22:35:16.340 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:16.336+0000 7f865d719a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:16.341 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:16.336+0000 7f865d719a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:16.342 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:16.336+0000 7f865d719a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:16.343 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:16.336+0000 7f865d719a40 -1 bdev(0x55cdd5914400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:35:16.343 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:16.336+0000 7f865d719a40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:35:16.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:35:16.963 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:35:16.964 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:35:16.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:35:16.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:35:17.256 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:35:17.257 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:35:17.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:35:17.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:35:17.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:35:17.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:35:17.279 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:17.272+0000 7fd11abf1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:17.281 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:17.276+0000 7fd11abf1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:17.282 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:17.276+0000 7fd11abf1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:17.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:35:17.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:35:17.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:35:17.464 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:35:17.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:35:17.465 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:17.465 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:35:17.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:35:17.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:17.466 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:35:17.641 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:17.636+0000 7fd11abf1a40 -1 Falling back to public interface 2026-02-25T22:35:17.682 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:18.329 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:18.324+0000 7fd11abf1a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:35:18.683 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:18.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:18.684 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:35:18.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:35:18.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:18.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:35:18.889 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:19.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:19.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:19.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:35:19.891 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:35:19.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:19.892 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:35:20.104 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/501133392,v1:127.0.0.1:6811/501133392] [v2:127.0.0.1:6812/501133392,v1:127.0.0.1:6813/501133392] exists,up 3cda0b6b-55fe-47b0-beda-a7288d10b693 2026-02-25T22:35:20.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:35:20.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:35:20.107 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:35:20.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:35:20.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:35:20.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:35:20.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:35:20.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:35:20.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:35:20.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:35:20.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:35:20.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:35:20.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:35:20.108 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:20.109 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:35:20.110 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:35:20.111 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:35:20.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a5c6a108-024a-4510-9ddd-4e5d147064a3 2026-02-25T22:35:20.111 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 a5c6a108-024a-4510-9ddd-4e5d147064a3 2026-02-25T22:35:20.112 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 a5c6a108-024a-4510-9ddd-4e5d147064a3' 2026-02-25T22:35:20.112 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:35:20.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAoeZ9pJT8oBxAA0ld7COLt7Qt6djH7bwsJxg== 2026-02-25T22:35:20.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAoeZ9pJT8oBxAA0ld7COLt7Qt6djH7bwsJxg=="}' 2026-02-25T22:35:20.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a5c6a108-024a-4510-9ddd-4e5d147064a3 -i td/test-erasure-eio/2/new.json 2026-02-25T22:35:20.344 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:35:20.353 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:35:20.354 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoeZ9pJT8oBxAA0ld7COLt7Qt6djH7bwsJxg== --osd-uuid a5c6a108-024a-4510-9ddd-4e5d147064a3 2026-02-25T22:35:20.371 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:20.368+0000 7f020f1cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:20.373 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:20.368+0000 7f020f1cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:20.374 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:20.368+0000 7f020f1cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:20.375 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:20.368+0000 7f020f1cda40 -1 bdev(0x55fad446c400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:35:20.375 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:20.368+0000 7f020f1cda40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:35:21.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:35:21.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:35:21.015 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:35:21.016 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:35:21.016 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:35:21.309 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:35:21.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:35:21.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:35:21.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:35:21.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:35:21.310 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:35:21.332 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:21.328+0000 7f826bae5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:21.334 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:21.328+0000 7f826bae5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:21.335 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:21.332+0000 7f826bae5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:21.522 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:35:21.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:35:21.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:35:21.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:35:21.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:35:21.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:21.524 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:35:21.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:35:21.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:21.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:35:21.725 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:21.720+0000 7f826bae5a40 -1 Falling back to public interface 2026-02-25T22:35:21.735 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:22.389 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:22.384+0000 7f826bae5a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:35:22.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:22.736 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:22.736 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:35:22.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:35:22.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:22.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:35:22.946 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:23.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:23.948 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:23.948 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:35:23.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:35:23.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:23.949 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:35:24.159 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1525999112,v1:127.0.0.1:6819/1525999112] [v2:127.0.0.1:6820/1525999112,v1:127.0.0.1:6821/1525999112] exists,up a5c6a108-024a-4510-9ddd-4e5d147064a3 2026-02-25T22:35:24.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:35:24.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:35:24.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:35:24.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:35:24.163 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:35:24.164 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:35:24.165 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:35:24.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:35:24.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:35:24.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:35:24.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:35:24.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:35:24.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:35:24.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:35:24.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:35:24.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:35:24.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:35:24.166 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:35:24.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ca440b6f-2644-44db-8cc8-de1cbbf62828 2026-02-25T22:35:24.167 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 ca440b6f-2644-44db-8cc8-de1cbbf62828' 2026-02-25T22:35:24.167 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 ca440b6f-2644-44db-8cc8-de1cbbf62828 2026-02-25T22:35:24.168 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:35:24.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAseZ9pIIJ3ChAAEe9LvuiHt/e4fd+kPy27fw== 2026-02-25T22:35:24.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAseZ9pIIJ3ChAAEe9LvuiHt/e4fd+kPy27fw=="}' 2026-02-25T22:35:24.177 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ca440b6f-2644-44db-8cc8-de1cbbf62828 -i td/test-erasure-eio/3/new.json 2026-02-25T22:35:24.391 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:35:24.400 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:35:24.401 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAseZ9pIIJ3ChAAEe9LvuiHt/e4fd+kPy27fw== --osd-uuid ca440b6f-2644-44db-8cc8-de1cbbf62828 2026-02-25T22:35:24.417 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:24.412+0000 7f9f8eca2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:24.418 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:24.412+0000 7f9f8eca2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:24.419 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:24.416+0000 7f9f8eca2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:24.420 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:24.416+0000 7f9f8eca2a40 -1 bdev(0x55a242dd6400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:35:24.420 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:24.416+0000 7f9f8eca2a40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:35:25.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:35:25.058 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:35:25.059 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:35:25.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:35:25.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:35:25.343 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:35:25.343 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:35:25.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:35:25.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:35:25.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:35:25.344 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:35:25.367 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:25.360+0000 7fe6066e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:25.368 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:25.364+0000 7fe6066e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:25.369 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:25.364+0000 7fe6066e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:25.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:35:25.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:35:25.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:35:25.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:35:25.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:35:25.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:25.555 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:35:25.556 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:35:25.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:25.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:35:25.764 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:25.789 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:25.784+0000 7fe6066e1a40 -1 Falling back to public interface 2026-02-25T22:35:26.369 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:26.364+0000 7fe6066e1a40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:35:26.766 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:26.766 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:35:26.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:26.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:35:26.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:26.767 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:35:26.979 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:27.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:27.981 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:27.981 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:35:27.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:35:27.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:27.982 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:35:28.184 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 24 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/709389902,v1:127.0.0.1:6827/709389902] [v2:127.0.0.1:6828/709389902,v1:127.0.0.1:6829/709389902] exists,up ca440b6f-2644-44db-8cc8-de1cbbf62828 2026-02-25T22:35:28.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:35:28.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:35:28.185 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:35:28.185 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:35:28.185 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:35:28.186 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:35:28.186 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:35:28.187 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:28.187 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:28.187 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:35:28.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:35:28.187 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:35:28.248 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:35:28.254 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:35:28.256 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:35:13.836+0000 7f3f0ab33a40 0 load: jerasure load: lrc 2026-02-25T22:35:28.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:355: TEST_rados_get_with_subreadall_eio_shard_0: local poolname=pool-jerasure 2026-02-25T22:35:28.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:356: TEST_rados_get_with_subreadall_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T22:35:28.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:35:28.257 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:35:28.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T22:35:28.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:35:28.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T22:35:28.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:35:28.258 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T22:35:28.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:35:28.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:35:28.890 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:35:28.900 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:35:29.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:35:29.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:35:29.901 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:35:29.902 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:35:29.902 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:35:29.903 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:35:29.903 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:35:29.903 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:35:29.904 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:35:29.904 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:35:30.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:35:30.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:35:30.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:35:30.007 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:35:30.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:35:30.008 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:35:30.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:35:30.209 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:35:30.209 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:35:30.209 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:35:30.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:35:30.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:30.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:35:30.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803781 2026-02-25T22:35:30.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803781 2026-02-25T22:35:30.288 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781' 2026-02-25T22:35:30.289 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:30.289 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:35:30.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607557 2026-02-25T22:35:30.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607557 2026-02-25T22:35:30.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607557' 2026-02-25T22:35:30.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:30.366 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:35:30.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444036 2026-02-25T22:35:30.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444036 2026-02-25T22:35:30.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607557 2-73014444036' 2026-02-25T22:35:30.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:30.446 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:35:30.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215107 2026-02-25T22:35:30.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215107 2026-02-25T22:35:30.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607557 2-73014444036 3-103079215107' 2026-02-25T22:35:30.524 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:30.524 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803781 2026-02-25T22:35:30.525 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:30.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:35:30.526 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803781 2026-02-25T22:35:30.526 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:30.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803781 2026-02-25T22:35:30.528 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803781' 2026-02-25T22:35:30.528 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803781 2026-02-25T22:35:30.529 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:35:30.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803779 -lt 25769803781 2026-02-25T22:35:30.739 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:35:31.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:35:31.741 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:35:31.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803781 2026-02-25T22:35:31.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:31.953 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607557 2026-02-25T22:35:31.953 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:31.954 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:35:31.955 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607557 2026-02-25T22:35:31.955 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:31.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607557 2026-02-25T22:35:31.957 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607557' 2026-02-25T22:35:31.957 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 51539607557 2026-02-25T22:35:31.957 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:35:32.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607557 -lt 51539607557 2026-02-25T22:35:32.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:32.170 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444036 2026-02-25T22:35:32.170 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:32.172 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:35:32.172 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444036 2026-02-25T22:35:32.173 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:32.174 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444036 2026-02-25T22:35:32.174 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 73014444036 2026-02-25T22:35:32.175 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444036' 2026-02-25T22:35:32.175 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:35:32.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444036 -lt 73014444036 2026-02-25T22:35:32.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:32.385 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-103079215107 2026-02-25T22:35:32.385 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:32.387 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:35:32.387 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-103079215107 2026-02-25T22:35:32.388 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:32.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=103079215107 2026-02-25T22:35:32.389 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 103079215107 2026-02-25T22:35:32.389 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 103079215107' 2026-02-25T22:35:32.390 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:35:32.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 103079215107 -lt 103079215107 2026-02-25T22:35:32.601 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:35:32.602 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:35:32.602 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:35:32.879 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:35:32.880 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:35:32.881 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:35:32.881 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:35:32.881 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:35:32.881 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:35:32.881 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:35:32.881 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:35:33.091 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:35:33.092 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:35:33.092 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:35:33.092 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:35:33.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:35:33.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:35:33.364 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:358: TEST_rados_get_with_subreadall_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 recovery 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-73931-0 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-73931-0 2026-02-25T22:35:33.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:35:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:35:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-73931-0 2026-02-25T22:35:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:35:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:35:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:35:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:35:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:35:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:35:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:35:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:35:33.366 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-73931-0 td/test-erasure-eio/ORIGINAL 2026-02-25T22:35:33.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-73931-0 td/test-erasure-eio 0 2026-02-25T22:35:33.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:35:33.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:35:33.383 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-73931-0 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-73931-0 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:35:33.384 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-73931-0 2026-02-25T22:35:33.385 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-73931-0 2026-02-25T22:35:33.385 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:35:33.600 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:35:33.600 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:35:33.600 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:35:33.600 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:35:33.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T22:35:33.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:35:33.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T22:35:33.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:35:33.601 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T22:35:33.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:35:33.602 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T22:35:33.603 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:35:33.603 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T22:35:33.603 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:35:33.603 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:35:33.603 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:35:33.604 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T22:35:33.604 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:35:33.604 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:35:33.604 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:35:33.604 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:33.604 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:33.605 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:35:33.605 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:35:33.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:35:33.666 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:35:33.667 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:35:33.668 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T22:35:33.668 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T22:35:33.668 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T22:35:33.668 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:35:33.668 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:33.668 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:33.669 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.3.asok 2026-02-25T22:35:33.669 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:35:33.669 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-73931-0 0 2026-02-25T22:35:33.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-73931-0 2026-02-25T22:35:33.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:35:33.727 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:35:33.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-73931-0 2026-02-25T22:35:33.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:35:33.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:35:33.728 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-73931-0 td/test-erasure-eio/COPY 2026-02-25T22:35:33.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:35:33.747 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:35:33.749 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2026-02-25T22:35:33.749 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-73931-0 2026-02-25T22:35:33.749 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:35:33.750 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-73931-0 2026-02-25T22:35:33.750 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-73931-0 2026-02-25T22:35:33.750 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:35:33.963 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:35:33.964 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:35:33.964 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:35:33.964 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:35:33.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2026-02-25T22:35:33.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2026-02-25T22:35:33.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2026-02-25T22:35:33.964 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T22:35:33.964 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:35:33.964 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:35:33.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:35:33.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:35:33.965 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:35:34.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:35:34.273 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2026-02-25T22:35:34.513 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 is already out. 2026-02-25T22:35:34.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-73931-0 2026-02-25T22:35:34.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:35:34.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-73931-0 2026-02-25T22:35:34.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2026-02-25T22:35:34.524 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-73931-0 2026-02-25T22:35:34.524 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:35:34.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:35:34.745 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:35:34.745 INFO:tasks.workunit.client.0.trial125.stderr:0' 2026-02-25T22:35:34.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 0 2026-02-25T22:35:34.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2026-02-25T22:35:35.006 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 is already in. 2026-02-25T22:35:35.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2026-02-25T22:35:35.017 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:35:35.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:35:35.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T22:35:35.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:35:35.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:35:35.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:35:35.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:35:35.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:35:35.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:35:35.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:35:35.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:35:35.018 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:35:35.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:35:35.019 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:35:35.019 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:35:35.019 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:35.019 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:35.019 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:35.019 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:35.019 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:35.019 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:35.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:35.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:35:35.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:35:35.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:35:35.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:35:35.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:35:35.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:35:35.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:35:35.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:35:35.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:35:35.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:35:35.022 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:35:35.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T22:35:35.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:35:35.023 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T22:35:35.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T22:35:35.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:35:35.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:35:35.023 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:35:35.042 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:35.036+0000 7fe57ada7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:35.043 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:35.040+0000 7fe57ada7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:35.044 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:35.040+0000 7fe57ada7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:35.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T22:35:35.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:35:35.233 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:35:35.234 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:35:35.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:35:35.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:35:35.234 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:35.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:35:35.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:35.235 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:35:35.446 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:35.493 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:35.488+0000 7fe57ada7a40 -1 Falling back to public interface 2026-02-25T22:35:36.195 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:36.188+0000 7fe57ada7a40 -1 osd.2 31 log_to_monitors true 2026-02-25T22:35:36.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:36.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:36.448 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:35:36.448 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:35:36.449 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:36.449 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:35:36.657 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 40 up_thru 17 down_at 32 last_clean_interval [17,31) [v2:127.0.0.1:6818/1842205039,v1:127.0.0.1:6819/1842205039] [v2:127.0.0.1:6820/1842205039,v1:127.0.0.1:6821/1842205039] exists,up a5c6a108-024a-4510-9ddd-4e5d147064a3 2026-02-25T22:35:36.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:35:36.658 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:35:36.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:35:36.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2026-02-25T22:35:36.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:35:36.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:35:36.659 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:35:36.659 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:35:36.659 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:35:36.659 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:35:36.660 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:35:36.660 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:35:36.660 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:35:36.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:35:36.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:35:36.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:35:36.761 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:35:36.762 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:35:36.762 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:35:36.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:35:36.967 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:35:36.967 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:35:36.967 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:35:36.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:35:36.967 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:36.968 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:35:37.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803785 2026-02-25T22:35:37.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803785 2026-02-25T22:35:37.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785' 2026-02-25T22:35:37.045 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:37.045 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:35:37.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607560 2026-02-25T22:35:37.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607560 2026-02-25T22:35:37.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-51539607560' 2026-02-25T22:35:37.122 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:37.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:35:37.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691843 2026-02-25T22:35:37.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691843 2026-02-25T22:35:37.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-51539607560 2-171798691843' 2026-02-25T22:35:37.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:35:37.202 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:35:37.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215110 2026-02-25T22:35:37.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215110 2026-02-25T22:35:37.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-51539607560 2-171798691843 3-103079215110' 2026-02-25T22:35:37.281 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:37.281 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803785 2026-02-25T22:35:37.282 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:37.283 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:35:37.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803785 2026-02-25T22:35:37.284 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:37.285 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803785 2026-02-25T22:35:37.285 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803785 2026-02-25T22:35:37.286 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2026-02-25T22:35:37.286 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:35:37.504 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803785 2026-02-25T22:35:37.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:35:38.505 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:35:38.506 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:35:38.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803785 2026-02-25T22:35:38.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:38.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607560 2026-02-25T22:35:38.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:38.721 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:35:38.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607560 2026-02-25T22:35:38.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:38.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607560 2026-02-25T22:35:38.724 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 51539607560 2026-02-25T22:35:38.725 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607560' 2026-02-25T22:35:38.725 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:35:38.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607560 -lt 51539607560 2026-02-25T22:35:38.931 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:38.931 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-171798691843 2026-02-25T22:35:38.932 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:38.933 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:35:38.934 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-171798691843 2026-02-25T22:35:38.934 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:38.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691843 2026-02-25T22:35:38.936 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 171798691843 2026-02-25T22:35:38.936 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 171798691843' 2026-02-25T22:35:38.936 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:35:39.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691843 -lt 171798691843 2026-02-25T22:35:39.150 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:35:39.150 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-103079215110 2026-02-25T22:35:39.151 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:35:39.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:35:39.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-103079215110 2026-02-25T22:35:39.153 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:35:39.154 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=103079215110 2026-02-25T22:35:39.154 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 103079215110 2026-02-25T22:35:39.155 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 103079215110' 2026-02-25T22:35:39.155 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:35:39.365 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 103079215110 -lt 103079215110 2026-02-25T22:35:39.365 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:35:39.366 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:35:39.366 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:35:39.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:35:39.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:35:39.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:35:39.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:35:39.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:35:39.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:35:39.646 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:35:39.646 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:35:39.849 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-25T22:35:39.850 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:35:39.850 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:35:39.851 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:35:40.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-25T22:35:40.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' -1 2026-02-25T22:35:40.139 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-25T22:35:40.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=3 2026-02-25T22:35:40.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:35:40.140 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-25T22:35:40.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:35:40.240 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:35:40.241 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:35:40.241 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:35:40.241 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:35:40.241 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:35:40.241 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:35:40.241 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:35:40.450 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-25T22:35:40.451 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:35:40.451 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:35:40.452 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:35:40.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-25T22:35:40.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-25T22:35:40.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:35:40.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:35:40.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:35:40.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:35:40.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:35:40.724 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:35:40.724 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:35:41.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:35:41.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:35:41.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-25T22:35:41.001 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:35:41.002 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-25T22:35:41.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:35:41.202 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:35:41.203 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:35:41.203 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:35:41.203 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:35:41.203 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:35:41.204 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:35:41.204 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:35:41.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-25T22:35:41.411 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:35:41.412 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:35:41.412 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:35:41.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-25T22:35:41.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-25T22:35:41.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:35:41.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:35:41.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:35:41.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:35:41.691 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:35:41.692 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:35:41.692 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:35:41.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:35:41.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:35:41.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-25T22:35:41.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:35:41.973 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-25T22:35:42.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:35:42.374 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:35:42.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:35:42.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:35:42.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:35:42.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:35:42.375 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:35:42.376 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:35:42.583 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:35:42.584 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:35:42.584 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:35:42.584 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:35:42.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:35:42.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:35:42.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:35:42.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_rados_get_with_subreadall_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:35:42.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:35:42.863 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:35:43.071 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:35:43.081 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:35:43.357 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:35:43.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:35:43.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:35:43.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:35:43.367 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:35:43.368 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:35:43.368 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:35:43.368 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:35:43.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:35:43.369 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:35:43.679 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:35:43.680 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:35:43.681 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:35:43.682 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:35:43.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:35:43.684 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:35:43.684 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:35:43.685 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:35:43.686 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:35:43.686 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:35:43.686 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:35:43.687 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:35:43.688 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:35:43.689 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:35:43.702 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:35:43.703 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:43.703 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:43.703 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:35:43.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:35:43.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:35:43.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T22:35:43.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T22:35:43.705 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T22:35:43.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T22:35:43.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:35:43.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:35:43.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:35:43.706 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:35:43.706 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:35:43.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:35:43.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:35:43.706 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:35:43.708 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:35:43.708 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:35:43.709 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:35:43.709 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:35:43.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:35:43.710 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:35:43.711 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:35:43.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:35:43.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:35:43.712 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:35:43.713 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:35:43.714 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:35:43.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:35:43.715 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:35:43.717 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:35:43.717 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:43.717 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:43.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:35:43.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:35:43.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:35:43.718 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T22:35:43.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:35:43.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:43.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:43.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73931 2026-02-25T22:35:43.722 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:35:43.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:35:43.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:35:43.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T22:35:43.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T22:35:43.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T22:35:43.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:35:43.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:35:43.723 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:35:43.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T22:35:43.724 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T22:35:43.745 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:35:43.745 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:43.745 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:43.746 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:43.746 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:43.746 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:43.746 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:43.746 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:35:43.774 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:35:43.775 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:35:43.775 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:35:43.775 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:35:43.775 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:35:43.776 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:35:43.776 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:35:43.777 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:35:43.777 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:35:43.777 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:35:43.777 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:43.777 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:43.777 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:35:43.778 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:35:43.778 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get fsid 2026-02-25T22:35:43.833 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:35:43.833 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:35:43.834 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:35:43.834 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:35:43.835 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:35:43.835 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:35:43.835 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:35:43.835 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:35:43.835 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:35:43.835 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:43.835 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:43.836 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:35:43.836 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:35:43.836 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73931/ceph-mon.a.asok config get mon_host 2026-02-25T22:35:43.890 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T22:35:43.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T22:35:43.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:35:43.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:35:43.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:35:43.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T22:35:43.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:35:44.000 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:35:44.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:44.001 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:44.001 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:44.001 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:44.002 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:44.002 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:44.002 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:35:44.003 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:35:44.021 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T22:35:44.022 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:35:44.134 INFO:tasks.workunit.client.0.trial125.stderr:pool 'rbd' already exists 2026-02-25T22:35:44.143 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:35:45.146 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T22:35:45.146 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:35:45.146 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:35:45.146 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:35:45.146 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:45.146 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:45.146 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-mon.a.asok 2026-02-25T22:35:45.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T22:35:45.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-mon.a.asok log flush 2026-02-25T22:35:45.195 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:35:45.201 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T22:35:45.203 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:35:43.760+0000 7fdf3e0b4ec0 0 load: jerasure load: lrc 2026-02-25T22:35:45.204 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_1 td/test-erasure-eio 2026-02-25T22:35:45.204 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_rados_get_with_subreadall_eio_shard_1: local dir=td/test-erasure-eio 2026-02-25T22:35:45.204 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_rados_get_with_subreadall_eio_shard_1: local shard_id=1 2026-02-25T22:35:45.204 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_rados_get_with_subreadall_eio_shard_1: setup_osds 4 2026-02-25T22:35:45.204 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T22:35:45.204 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T22:35:45.204 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T22:35:45.205 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T22:35:45.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:35:45.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T22:35:45.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:35:45.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:35:45.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:35:45.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:35:45.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T22:35:45.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:35:45.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:35:45.209 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:45.210 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:35:45.211 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:35:45.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T22:35:45.212 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:35:45.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=04cb8101-5819-48a9-ab20-a4d557ceb36c 2026-02-25T22:35:45.212 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 04cb8101-5819-48a9-ab20-a4d557ceb36c' 2026-02-25T22:35:45.213 INFO:tasks.workunit.client.0.trial125.stdout:add osd0 04cb8101-5819-48a9-ab20-a4d557ceb36c 2026-02-25T22:35:45.213 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:35:45.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBeZ9pJ30eDRAANQISCkJL8jwhCBcN9YmAtw== 2026-02-25T22:35:45.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBeZ9pJ30eDRAANQISCkJL8jwhCBcN9YmAtw=="}' 2026-02-25T22:35:45.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 04cb8101-5819-48a9-ab20-a4d557ceb36c -i td/test-erasure-eio/0/new.json 2026-02-25T22:35:45.332 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:35:45.339 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T22:35:45.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBeZ9pJ30eDRAANQISCkJL8jwhCBcN9YmAtw== --osd-uuid 04cb8101-5819-48a9-ab20-a4d557ceb36c 2026-02-25T22:35:45.356 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:45.352+0000 7f86b01faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:45.358 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:45.352+0000 7f86b01faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:45.359 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:45.352+0000 7f86b01faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:45.359 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:45.356+0000 7f86b01faa40 -1 bdev(0x5560de108400 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:35:45.360 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:45.356+0000 7f86b01faa40 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T22:35:46.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T22:35:46.030 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:35:46.031 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:35:46.031 INFO:tasks.workunit.client.0.trial125.stdout:adding osd0 key to auth repository 2026-02-25T22:35:46.032 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:35:46.147 INFO:tasks.workunit.client.0.trial125.stdout:start osd.0 2026-02-25T22:35:46.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:35:46.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:35:46.148 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:35:46.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:35:46.170 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:35:46.210 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:46.200+0000 7f07ff6aaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:46.211 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:46.204+0000 7f07ff6aaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:46.212 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:46.208+0000 7f07ff6aaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:46.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:35:46.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:35:46.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:35:46.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:35:46.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:35:46.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:46.301 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:35:46.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:35:46.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:46.301 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:35:46.408 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:46.605 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:46.600+0000 7f07ff6aaa40 -1 Falling back to public interface 2026-02-25T22:35:47.184 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:47.180+0000 7f07ff6aaa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:35:47.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:47.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:47.410 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:35:47.410 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:35:47.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:47.411 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:35:47.636 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:48.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:48.638 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:48.639 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:35:48.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:35:48.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:48.639 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:35:48.846 INFO:tasks.workunit.client.0.trial125.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1396071561,v1:127.0.0.1:6803/1396071561] [v2:127.0.0.1:6804/1396071561,v1:127.0.0.1:6805/1396071561] exists,up 04cb8101-5819-48a9-ab20-a4d557ceb36c 2026-02-25T22:35:48.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:35:48.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:35:48.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:35:48.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:35:48.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T22:35:48.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:35:48.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:35:48.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:35:48.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:35:48.850 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:48.851 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:35:48.852 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:35:48.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:35:48.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:35:48.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:35:48.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T22:35:48.853 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:35:48.853 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ef51823b-1157-4b19-a05d-528e45b88f66 2026-02-25T22:35:48.854 INFO:tasks.workunit.client.0.trial125.stdout:add osd1 ef51823b-1157-4b19-a05d-528e45b88f66 2026-02-25T22:35:48.854 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ef51823b-1157-4b19-a05d-528e45b88f66' 2026-02-25T22:35:48.854 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:35:48.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBEeZ9pJ9RTMxAAMGNuTawqwX21pyo/se6JJA== 2026-02-25T22:35:48.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBEeZ9pJ9RTMxAAMGNuTawqwX21pyo/se6JJA=="}' 2026-02-25T22:35:48.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ef51823b-1157-4b19-a05d-528e45b88f66 -i td/test-erasure-eio/1/new.json 2026-02-25T22:35:49.076 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:35:49.086 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T22:35:49.087 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEeZ9pJ9RTMxAAMGNuTawqwX21pyo/se6JJA== --osd-uuid ef51823b-1157-4b19-a05d-528e45b88f66 2026-02-25T22:35:49.103 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:49.100+0000 7f8f34762a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:49.105 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:49.100+0000 7f8f34762a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:49.106 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:49.100+0000 7f8f34762a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:49.106 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:49.100+0000 7f8f34762a40 -1 bdev(0x559a974f4400 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:35:49.107 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:49.100+0000 7f8f34762a40 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T22:35:49.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T22:35:49.768 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:35:49.769 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:35:49.770 INFO:tasks.workunit.client.0.trial125.stdout:adding osd1 key to auth repository 2026-02-25T22:35:49.770 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:35:50.055 INFO:tasks.workunit.client.0.trial125.stdout:start osd.1 2026-02-25T22:35:50.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:35:50.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:35:50.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:35:50.056 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:35:50.057 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:35:50.071 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:50.064+0000 7fe46d565a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:50.072 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:50.068+0000 7fe46d565a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:50.074 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:50.068+0000 7fe46d565a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:50.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:35:50.261 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:35:50.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:35:50.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:35:50.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:35:50.262 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:50.262 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:35:50.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:35:50.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:50.263 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:35:50.478 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:50.581 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:50.576+0000 7fe46d565a40 -1 Falling back to public interface 2026-02-25T22:35:51.176 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:51.172+0000 7fe46d565a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:35:51.479 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:51.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:51.480 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:35:51.480 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:35:51.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:51.481 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:35:51.700 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:52.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:52.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:52.701 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:35:52.702 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:35:52.702 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:52.702 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:35:52.906 INFO:tasks.workunit.client.0.trial125.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2269511849,v1:127.0.0.1:6811/2269511849] [v2:127.0.0.1:6812/2269511849,v1:127.0.0.1:6813/2269511849] exists,up ef51823b-1157-4b19-a05d-528e45b88f66 2026-02-25T22:35:52.909 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:35:52.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:35:52.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:35:52.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:35:52.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T22:35:52.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:35:52.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:35:52.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:35:52.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:35:52.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:35:52.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:35:52.910 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:52.911 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:35:52.912 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:35:52.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:35:52.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:35:52.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:35:52.913 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:35:52.913 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=eed8a026-76a7-47eb-8c48-668a97dc63c8 2026-02-25T22:35:52.914 INFO:tasks.workunit.client.0.trial125.stdout:add osd2 eed8a026-76a7-47eb-8c48-668a97dc63c8 2026-02-25T22:35:52.914 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 eed8a026-76a7-47eb-8c48-668a97dc63c8' 2026-02-25T22:35:52.914 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:35:52.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBIeZ9pUjwQNxAAghobU/T/eimCl6R0LHhhSw== 2026-02-25T22:35:52.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBIeZ9pUjwQNxAAghobU/T/eimCl6R0LHhhSw=="}' 2026-02-25T22:35:52.925 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new eed8a026-76a7-47eb-8c48-668a97dc63c8 -i td/test-erasure-eio/2/new.json 2026-02-25T22:35:53.141 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:35:53.151 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T22:35:53.152 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIeZ9pUjwQNxAAghobU/T/eimCl6R0LHhhSw== --osd-uuid eed8a026-76a7-47eb-8c48-668a97dc63c8 2026-02-25T22:35:53.168 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:53.164+0000 7f19889a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:53.170 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:53.164+0000 7f19889a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:53.171 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:53.164+0000 7f19889a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:53.171 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:53.164+0000 7f19889a6a40 -1 bdev(0x561c537d2400 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:35:53.171 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:53.168+0000 7f19889a6a40 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T22:35:53.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T22:35:53.818 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:35:53.819 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:35:53.820 INFO:tasks.workunit.client.0.trial125.stdout:adding osd2 key to auth repository 2026-02-25T22:35:53.820 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:35:54.104 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:35:54.105 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:35:54.105 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:35:54.105 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:35:54.106 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:35:54.106 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:35:54.127 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:54.120+0000 7f9b4a3cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:54.129 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:54.124+0000 7f9b4a3cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:54.130 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:54.124+0000 7f9b4a3cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:54.314 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:35:54.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:35:54.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:35:54.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:35:54.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:35:54.315 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:54.315 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:35:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:35:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:54.316 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:35:54.465 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:54.460+0000 7f9b4a3cda40 -1 Falling back to public interface 2026-02-25T22:35:54.523 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:55.065 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:55.060+0000 7f9b4a3cda40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:35:55.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:55.525 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:55.525 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:35:55.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:35:55.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:55.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:35:55.736 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2798572477,v1:127.0.0.1:6819/2798572477] [v2:127.0.0.1:6820/2798572477,v1:127.0.0.1:6821/2798572477] exists,up eed8a026-76a7-47eb-8c48-668a97dc63c8 2026-02-25T22:35:55.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:35:55.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:35:55.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:35:55.740 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T22:35:55.741 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:35:55.742 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:35:55.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:35:55.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:35:55.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:35:55.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:35:55.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:35:55.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:35:55.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:35:55.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:35:55.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:35:55.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:35:55.743 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T22:35:55.744 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:35:55.744 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0cc6b921-c14e-440f-996b-86c67cb9246b 2026-02-25T22:35:55.744 INFO:tasks.workunit.client.0.trial125.stdout:add osd3 0cc6b921-c14e-440f-996b-86c67cb9246b 2026-02-25T22:35:55.745 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 0cc6b921-c14e-440f-996b-86c67cb9246b' 2026-02-25T22:35:55.745 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:35:55.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBLeZ9pSrHfLBAANCRd84i36AhLK0kdN5BvwA== 2026-02-25T22:35:55.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBLeZ9pSrHfLBAANCRd84i36AhLK0kdN5BvwA=="}' 2026-02-25T22:35:55.754 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0cc6b921-c14e-440f-996b-86c67cb9246b -i td/test-erasure-eio/3/new.json 2026-02-25T22:35:55.966 INFO:tasks.workunit.client.0.trial125.stdout:3 2026-02-25T22:35:55.975 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T22:35:55.976 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLeZ9pSrHfLBAANCRd84i36AhLK0kdN5BvwA== --osd-uuid 0cc6b921-c14e-440f-996b-86c67cb9246b 2026-02-25T22:35:55.992 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:55.988+0000 7f7bfbd99a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:55.994 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:55.988+0000 7f7bfbd99a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:55.995 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:55.988+0000 7f7bfbd99a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:55.995 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:55.988+0000 7f7bfbd99a40 -1 bdev(0x5575ad926400 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:35:55.995 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:55.992+0000 7f7bfbd99a40 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T22:35:56.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T22:35:56.650 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:35:56.651 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:35:56.651 INFO:tasks.workunit.client.0.trial125.stdout:adding osd3 key to auth repository 2026-02-25T22:35:56.652 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:35:56.934 INFO:tasks.workunit.client.0.trial125.stdout:start osd.3 2026-02-25T22:35:56.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:35:56.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:35:56.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:35:56.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:35:56.935 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:35:56.958 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:56.952+0000 7f4572e0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:56.960 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:56.956+0000 7f4572e0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:56.961 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:56.956+0000 7f4572e0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:35:57.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:35:57.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:35:57.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:35:57.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:35:57.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:35:57.145 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:57.146 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:35:57.146 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:35:57.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:57.147 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:35:57.355 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:57.369 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:57.364+0000 7f4572e0da40 -1 Falling back to public interface 2026-02-25T22:35:57.958 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:35:57.952+0000 7f4572e0da40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:35:58.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:58.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:58.357 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:35:58.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:35:58.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:58.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:35:58.569 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:35:59.570 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:35:59.571 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:35:59.571 INFO:tasks.workunit.client.0.trial125.stdout:2 2026-02-25T22:35:59.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:35:59.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:35:59.572 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:35:59.780 INFO:tasks.workunit.client.0.trial125.stdout:osd.3 up in weight 1 up_from 26 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3527659604,v1:127.0.0.1:6827/3527659604] [v2:127.0.0.1:6828/3527659604,v1:127.0.0.1:6829/3527659604] exists,up 0cc6b921-c14e-440f-996b-86c67cb9246b 2026-02-25T22:35:59.780 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:35:59.780 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:35:59.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:35:59.782 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T22:35:59.782 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:35:59.782 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:35:59.782 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:35:59.782 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:35:59.782 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:35:59.782 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.0.asok 2026-02-25T22:35:59.783 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T22:35:59.783 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.0.asok log flush 2026-02-25T22:35:59.836 INFO:tasks.workunit.client.0.trial125.stdout:{} 2026-02-25T22:35:59.842 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T22:35:59.844 INFO:tasks.workunit.client.0.trial125.stdout:2026-02-25T22:35:46.604+0000 7f07ff6aaa40 0 load: jerasure load: lrc 2026-02-25T22:35:59.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_rados_get_with_subreadall_eio_shard_1: local poolname=pool-jerasure 2026-02-25T22:35:59.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:370: TEST_rados_get_with_subreadall_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T22:35:59.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:35:59.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T22:35:59.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T22:35:59.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T22:35:59.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T22:35:59.845 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T22:35:59.846 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T22:36:00.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T22:36:00.111 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T22:36:00.475 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' already exists 2026-02-25T22:36:00.485 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:36:01.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T22:36:01.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:36:01.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:36:01.486 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:36:01.487 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:36:01.488 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:36:01.488 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:36:01.488 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:36:01.488 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:36:01.489 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:36:01.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:36:01.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:36:01.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:36:01.600 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:36:01.601 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:36:01.601 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:36:01.804 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:36:01.804 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:36:01.804 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:36:01.805 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:36:01.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:36:01.805 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:36:01.805 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:36:01.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803781 2026-02-25T22:36:01.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803781 2026-02-25T22:36:01.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781' 2026-02-25T22:36:01.885 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:36:01.885 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:36:01.961 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607556 2026-02-25T22:36:01.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607556 2026-02-25T22:36:01.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607556' 2026-02-25T22:36:01.962 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:36:01.962 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:36:02.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411332 2026-02-25T22:36:02.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411332 2026-02-25T22:36:02.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607556 2-77309411332' 2026-02-25T22:36:02.041 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:36:02.041 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:36:02.119 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149699 2026-02-25T22:36:02.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149699 2026-02-25T22:36:02.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607556 2-77309411332 3-111669149699' 2026-02-25T22:36:02.120 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:36:02.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803781 2026-02-25T22:36:02.120 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:36:02.121 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:36:02.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803781 2026-02-25T22:36:02.122 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:36:02.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803781 2026-02-25T22:36:02.124 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803781' 2026-02-25T22:36:02.124 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803781 2026-02-25T22:36:02.125 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:36:02.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803781 2026-02-25T22:36:02.335 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:36:02.336 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607556 2026-02-25T22:36:02.336 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:36:02.337 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:36:02.338 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607556 2026-02-25T22:36:02.339 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:36:02.340 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607556 2026-02-25T22:36:02.340 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 51539607556 2026-02-25T22:36:02.341 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607556' 2026-02-25T22:36:02.341 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:36:02.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607556 -lt 51539607556 2026-02-25T22:36:02.553 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:36:02.554 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411332 2026-02-25T22:36:02.554 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:36:02.556 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:36:02.556 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411332 2026-02-25T22:36:02.557 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:36:02.558 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411332 2026-02-25T22:36:02.558 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 77309411332 2026-02-25T22:36:02.559 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411332' 2026-02-25T22:36:02.559 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:36:02.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411332 -lt 77309411332 2026-02-25T22:36:02.781 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:36:02.782 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149699 2026-02-25T22:36:02.782 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:36:02.783 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:36:02.784 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149699 2026-02-25T22:36:02.784 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:36:02.785 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149699 2026-02-25T22:36:02.786 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149699 2026-02-25T22:36:02.786 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149699' 2026-02-25T22:36:02.786 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:36:03.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149699 -lt 111669149699 2026-02-25T22:36:03.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:36:03.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:36:03.010 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:36:03.293 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:36:03.294 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:36:03.294 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:36:03.295 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:36:03.295 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:36:03.295 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:36:03.295 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:36:03.295 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:36:03.514 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:36:03.514 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:36:03.515 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:36:03.515 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:36:03.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T22:36:03.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-25T22:36:03.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-25T22:36:03.790 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-25T22:36:03.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:36:03.791 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-25T22:36:03.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:36:03.891 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:36:03.892 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:36:03.892 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:36:03.892 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:36:03.893 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:36:03.893 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:36:03.893 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:36:04.098 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:36:04.098 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:36:04.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:36:04.099 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:36:04.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T22:36:04.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-25T22:36:04.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T22:36:04.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:36:04.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:36:04.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:36:04.373 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:36:04.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:36:04.374 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:36:04.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:36:04.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:36:04.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-25T22:36:04.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T22:36:04.665 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-25T22:36:04.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T22:36:04.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:36:04.867 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:36:04.867 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:36:04.867 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:36:04.867 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:36:04.867 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:36:04.868 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:36:05.077 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:36:05.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:36:05.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:36:05.078 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:36:05.356 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:36:05.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:36:05.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:36:05.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_rados_get_with_subreadall_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 recovery 2026-02-25T22:36:05.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-25T22:36:05.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-25T22:36:05.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-25T22:36:05.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-25T22:36:05.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2026-02-25T22:36:05.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-25T22:36:05.357 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-73931-1 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-73931-1 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-73931-1 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T22:36:05.358 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T22:36:05.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T22:36:05.359 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-73931-1 td/test-erasure-eio/ORIGINAL 2026-02-25T22:36:05.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-73931-1 td/test-erasure-eio 1 2026-02-25T22:36:05.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T22:36:05.375 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-73931-1 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-73931-1 2026-02-25T22:36:05.376 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:36:05.377 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-73931-1 2026-02-25T22:36:05.377 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-73931-1 2026-02-25T22:36:05.377 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:36:05.589 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:36:05.589 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:36:05.589 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:36:05.589 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:36:05.589 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T22:36:05.589 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T22:36:05.589 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=1 2026-02-25T22:36:05.589 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T22:36:05.590 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/1/type 2026-02-25T22:36:05.590 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T22:36:05.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-25T22:36:05.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T22:36:05.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-25T22:36:05.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T22:36:05.591 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T22:36:05.592 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T22:36:05.592 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-25T22:36:05.592 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:36:05.592 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:36:05.593 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:36:05.593 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:36:05.593 INFO:tasks.workunit.client.0.trial125.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:36:05.593 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.1.asok 2026-02-25T22:36:05.593 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.73931/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-25T22:36:05.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T22:36:05.654 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T22:36:05.655 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T22:36:05.656 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.1 2026-02-25T22:36:05.656 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:36:05.656 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:36:05.657 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:36:05.657 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:36:05.657 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:36:05.657 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73931/ceph-osd.1.asok 2026-02-25T22:36:05.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T22:36:05.657 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.73931/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-73931-1 1 2026-02-25T22:36:05.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-73931-1 2026-02-25T22:36:05.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T22:36:05.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T22:36:05.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-73931-1 2026-02-25T22:36:05.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T22:36:05.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T22:36:05.717 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-73931-1 td/test-erasure-eio/COPY 2026-02-25T22:36:05.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T22:36:05.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T22:36:05.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2026-02-25T22:36:05.739 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-73931-1 2026-02-25T22:36:05.739 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:36:05.740 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-73931-1 2026-02-25T22:36:05.740 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-73931-1 2026-02-25T22:36:05.740 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:36:05.950 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:36:05.950 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:36:05.950 INFO:tasks.workunit.client.0.trial125.stderr:2' 2026-02-25T22:36:05.950 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T22:36:05.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2026-02-25T22:36:05.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2026-02-25T22:36:05.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2026-02-25T22:36:05.951 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T22:36:05.951 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:36:05.952 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:36:05.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:36:05.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:36:05.952 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:36:06.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:36:06.259 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2026-02-25T22:36:06.515 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 is already out. 2026-02-25T22:36:06.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-73931-1 2026-02-25T22:36:06.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T22:36:06.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-73931-1 2026-02-25T22:36:06.526 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2026-02-25T22:36:06.526 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-73931-1 2026-02-25T22:36:06.526 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:36:06.737 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T22:36:06.737 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:36:06.738 INFO:tasks.workunit.client.0.trial125.stderr:0' 2026-02-25T22:36:06.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 0 2026-02-25T22:36:06.738 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2026-02-25T22:36:06.996 INFO:tasks.workunit.client.0.trial125.stderr:osd.2 is already in. 2026-02-25T22:36:07.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2026-02-25T22:36:07.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T22:36:07.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:36:07.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T22:36:07.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:36:07.008 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:36:07.009 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:36:07.010 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:36:07.010 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:36:07.010 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:36:07.011 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:36:07.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' 2026-02-25T22:36:07.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:36:07.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T22:36:07.011 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T22:36:07.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:36:07.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:36:07.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:36:07.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:36:07.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:36:07.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:36:07.012 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T22:36:07.012 INFO:tasks.workunit.client.0.trial125.stdout:start osd.2 2026-02-25T22:36:07.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T22:36:07.013 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=4069c10d-68e8-4e38-a064-680cd7920277 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.73931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:36:07.014 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T22:36:07.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T22:36:07.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:36:07.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:36:07.014 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:36:07.028 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:36:07.023+0000 7fabedce3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:36:07.030 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:36:07.023+0000 7fabedce3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:36:07.031 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:36:07.027+0000 7fabedce3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:36:07.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T22:36:07.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:36:07.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:36:07.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:36:07.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:36:07.221 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:36:07.221 INFO:tasks.workunit.client.0.trial125.stdout:0 2026-02-25T22:36:07.222 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:36:07.222 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:36:07.222 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:36:07.389 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:36:07.383+0000 7fabedce3a40 -1 Falling back to public interface 2026-02-25T22:36:07.430 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:36:08.088 INFO:tasks.workunit.client.0.trial125.stderr:2026-02-25T22:36:08.083+0000 7fabedce3a40 -1 osd.2 34 log_to_monitors true 2026-02-25T22:36:08.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:36:08.431 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:36:08.432 INFO:tasks.workunit.client.0.trial125.stdout:1 2026-02-25T22:36:08.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:36:08.432 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:36:08.433 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:36:08.644 INFO:tasks.workunit.client.0.trial125.stdout:osd.2 up in weight 1 up_from 43 up_thru 20 down_at 35 last_clean_interval [18,34) [v2:127.0.0.1:6818/2205000260,v1:127.0.0.1:6819/2205000260] [v2:127.0.0.1:6820/2205000260,v1:127.0.0.1:6821/2205000260] exists,up eed8a026-76a7-47eb-8c48-668a97dc63c8 2026-02-25T22:36:08.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:36:08.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:36:08.644 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:36:08.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2026-02-25T22:36:08.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:36:08.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:36:08.645 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:36:08.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:36:08.645 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:36:08.645 INFO:tasks.workunit.client.0.trial125.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:36:08.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:36:08.645 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:36:08.646 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:36:08.757 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:36:08.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:36:08.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:36:08.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:36:08.758 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:36:08.758 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:36:08.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:36:08.978 INFO:tasks.workunit.client.0.trial125.stderr:1 2026-02-25T22:36:08.978 INFO:tasks.workunit.client.0.trial125.stderr:2 2026-02-25T22:36:08.978 INFO:tasks.workunit.client.0.trial125.stderr:3' 2026-02-25T22:36:08.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:36:08.978 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:36:08.978 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:36:09.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803785 2026-02-25T22:36:09.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803785 2026-02-25T22:36:09.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785' 2026-02-25T22:36:09.059 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:36:09.059 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:36:09.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607560 2026-02-25T22:36:09.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607560 2026-02-25T22:36:09.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-51539607560' 2026-02-25T22:36:09.137 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:36:09.137 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:36:09.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593731 2026-02-25T22:36:09.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593731 2026-02-25T22:36:09.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-51539607560 2-184683593731' 2026-02-25T22:36:09.216 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:36:09.216 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:36:09.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149703 2026-02-25T22:36:09.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149703 2026-02-25T22:36:09.295 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-51539607560 2-184683593731 3-111669149703' 2026-02-25T22:36:09.296 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:36:09.296 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803785 2026-02-25T22:36:09.296 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:36:09.297 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:36:09.298 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803785 2026-02-25T22:36:09.298 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:36:09.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803785 2026-02-25T22:36:09.300 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2026-02-25T22:36:09.300 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.0 seq 25769803785 2026-02-25T22:36:09.300 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:36:09.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803785 2026-02-25T22:36:09.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:36:10.512 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:36:10.513 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:36:10.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803785 2026-02-25T22:36:10.726 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:36:10.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607560 2026-02-25T22:36:10.727 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:36:10.729 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:36:10.729 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607560 2026-02-25T22:36:10.729 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:36:10.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607560 2026-02-25T22:36:10.731 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.1 seq 51539607560 2026-02-25T22:36:10.731 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607560' 2026-02-25T22:36:10.731 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:36:10.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607560 -lt 51539607560 2026-02-25T22:36:10.942 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:36:10.943 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-184683593731 2026-02-25T22:36:10.943 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:36:10.944 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:36:10.945 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-184683593731 2026-02-25T22:36:10.945 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:36:10.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593731 2026-02-25T22:36:10.947 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 184683593731' 2026-02-25T22:36:10.947 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.2 seq 184683593731 2026-02-25T22:36:10.947 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:36:11.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593731 -lt 184683593731 2026-02-25T22:36:11.166 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:36:11.167 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149703 2026-02-25T22:36:11.167 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:36:11.169 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:36:11.169 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149703 2026-02-25T22:36:11.169 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:36:11.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149703 2026-02-25T22:36:11.171 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149703' 2026-02-25T22:36:11.171 INFO:tasks.workunit.client.0.trial125.stdout:waiting osd.3 seq 111669149703 2026-02-25T22:36:11.171 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:36:11.385 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149703 -lt 111669149703 2026-02-25T22:36:11.386 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:36:11.386 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:36:11.386 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:36:11.661 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T22:36:11.661 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:36:11.662 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:36:11.662 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:36:11.662 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:36:11.662 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:36:11.662 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:36:11.662 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:36:11.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T22:36:11.871 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:36:11.872 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:36:11.872 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:36:12.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T22:36:12.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:36:12.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:36:12.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:374: TEST_rados_get_with_subreadall_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2026-02-25T22:36:12.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T22:36:12.159 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T22:36:12.428 INFO:tasks.workunit.client.0.trial125.stderr:pool 'pool-jerasure' does not exist 2026-02-25T22:36:12.437 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T22:36:12.708 INFO:tasks.workunit.client.0.trial125.stderr:erasure-code-profile myprofile does not exist 2026-02-25T22:36:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T22:36:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:36:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:36:12.719 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:36:12.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:36:12.720 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:36:12.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:36:12.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:36:12.720 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:36:12.832 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:36:12.833 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:36:12.834 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:36:12.834 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:36:12.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:36:12.836 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:36:12.836 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:36:12.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:36:12.837 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:36:12.838 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:36:12.838 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:36:12.839 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:36:12.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:36:12.840 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:36:12.855 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:36:12.855 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:36:12.855 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:36:12.855 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:36:12.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:36:12.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:36:12.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T22:36:12.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/test-erasure-eio 0 2026-02-25T22:36:12.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T22:36:12.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T22:36:12.857 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T22:36:12.857 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:36:12.857 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:36:12.858 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:36:12.858 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:36:12.858 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:36:12.860 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:36:12.861 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:36:12.862 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:36:12.862 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:36:12.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:36:12.864 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:36:12.864 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:36:12.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:36:12.865 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:36:12.866 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:36:12.866 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:36:12.867 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:36:12.868 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T22:36:12.869 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T22:36:12.870 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:36:12.870 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:36:12.870 INFO:tasks.workunit.client.0.trial125.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73931 2026-02-25T22:36:12.870 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73931 2026-02-25T22:36:12.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:36:12.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:36:12.871 INFO:tasks.workunit.client.0.trial125.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T22:36:12.872 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T22:36:12.872 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T22:36:12.926 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2026-02-25T22:36:12.926 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-25T22:36:13.291 DEBUG:teuthology.parallel:result is None 2026-02-25T22:36:13.291 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T22:36:13.299 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T22:36:13.299 DEBUG:teuthology.orchestra.run.trial125:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T22:36:13.342 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T22:36:13.342 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T22:36:13.355 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-25T22:36:13.355 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T22:36:13.447 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-25T22:36:13.457 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-25T22:36:13.457 DEBUG:teuthology.orchestra.run.trial125:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T22:36:13.520 INFO:teuthology.orchestra.run.trial125.stdout: remote refid st t when poll reach delay offset jitter 2026-02-25T22:36:13.520 INFO:teuthology.orchestra.run.trial125.stdout:============================================================================== 2026-02-25T22:36:13.520 INFO:teuthology.orchestra.run.trial125.stdout:-172-232-15-202. 81.104.22.229 2 u 6 64 377 27.304 +5.117 3.471 2026-02-25T22:36:13.520 INFO:teuthology.orchestra.run.trial125.stdout:*50.218.103.254 50.205.57.38 2 u 59 64 377 9.300 -2.124 0.278 2026-02-25T22:36:13.520 INFO:teuthology.orchestra.run.trial125.stdout:+time-h.den.code 132.163.97.5 2 u - 64 377 40.011 +1.978 9.188 2026-02-25T22:36:13.520 INFO:teuthology.orchestra.run.trial125.stdout:+172-104-28-175. 80.192.165.246 2 u - 64 377 3.337 +4.355 0.098 2026-02-25T22:36:13.521 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-25T22:36:13.532 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-25T22:36:13.532 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-25T22:36:13.546 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-25T22:36:13.559 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-25T22:36:13.572 INFO:teuthology.task.internal:Duration was 1139.190994 seconds 2026-02-25T22:36:13.572 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-25T22:36:13.586 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-25T22:36:13.587 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T22:36:13.610 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-25T22:36:13.610 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial125.front.sepia.ceph.com 2026-02-25T22:36:13.610 DEBUG:teuthology.orchestra.run.trial125:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T22:36:13.656 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-25T22:36:13.656 DEBUG:teuthology.orchestra.run.trial125:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T22:36:13.923 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-25T22:36:13.923 DEBUG:teuthology.orchestra.run.trial125:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T22:36:13.931 INFO:teuthology.orchestra.run.trial125.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T22:36:13.931 INFO:teuthology.orchestra.run.trial125.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T22:36:13.932 INFO:teuthology.orchestra.run.trial125.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T22:36:13.932 INFO:teuthology.orchestra.run.trial125.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T22:36:13.932 INFO:teuthology.orchestra.run.trial125.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T22:36:14.002 INFO:teuthology.orchestra.run.trial125.stderr: 81.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T22:36:14.003 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-25T22:36:14.014 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-25T22:36:14.014 DEBUG:teuthology.orchestra.run.trial125:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T22:36:14.054 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-25T22:36:14.064 DEBUG:teuthology.orchestra.run.trial125:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:36:14.102 INFO:teuthology.orchestra.run.trial125.stdout:kernel.core_pattern = core 2026-02-25T22:36:14.110 DEBUG:teuthology.orchestra.run.trial125:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:36:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T22:36:14.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-25T22:36:14.164 INFO:teuthology.task.internal:Transferring archived files... 2026-02-25T22:36:14.165 DEBUG:teuthology.misc:Transferring archived files from trial125:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/emzhang-2026-02-25_22:10:53-rados:standalone-wip-mon-colocate-distro-default-trial/70699/remote/trial125 2026-02-25T22:36:14.166 DEBUG:teuthology.orchestra.run.trial125:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T22:36:14.214 INFO:teuthology.task.internal:Removing archive directory... 2026-02-25T22:36:14.214 DEBUG:teuthology.orchestra.run.trial125:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T22:36:14.250 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-25T22:36:14.261 INFO:teuthology.task.internal:Not uploading archives. 2026-02-25T22:36:14.261 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-25T22:36:14.275 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-25T22:36:14.275 DEBUG:teuthology.orchestra.run.trial125:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T22:36:14.294 INFO:teuthology.orchestra.run.trial125.stdout: 1311292 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 22:36 /home/ubuntu/cephtest 2026-02-25T22:36:14.295 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-25T22:36:14.306 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-25T22:36:14.335 INFO:teuthology.run:Summary data: description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} duration: 1139.1909942626953 flavor: default owner: scheduled_emzhang@soko04.front.sepia.ceph.com success: true 2026-02-25T22:36:14.335 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T22:36:14.425 INFO:teuthology.run:pass